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FOREWORD 


In  a  1980  report  of  issues  identified  in  studies  of  21 
major  weapon  systems,  the  General  Accounting  Office  wrote: 

Durinq  weapon  system  development,  the  adequacy  of 
test  i  ng  is  critical  to  assess  and  reduce  program 
acquisition  risks  and  to  evaluate  operational 
effectiveness  and  suitability.  The  purpose  of  testing 
is  to  minimize  uncertainties  that  could  adversely 
affect  system  effectiveness,  cost,  or  availability  for 
deployment.  We  found  examples  of  testing  not  being 
comprehensive,  rigorous,  and  complete. 

The  phrase,  adequacy  of  testing,  has,  in  a  sense,  be¬ 
come  a  battle  cry  for  those  who  believe  that  the  Depart¬ 
ment  of  Defense  takes  unwarranted  risks  in  acquiring  its 
weapon  systems  and  moves  precipitately  toward  greater  and 
greater  commitment  to  unproven,  sometimes  unworkable,  de¬ 
signs.  On  the  other  hand,  some,  including  the  1977  De¬ 
fense  Science  Board,  are  concerned  that  current  policies 
encourage  overtesting  and  that  the  acquisition  cycle  may 
be  lengthened  because  "...the  independent  test  agencies 
will  gradually  gain  increased  influence  and  demand  dupli¬ 
cate  test  activities.” 

The  test  aaencies  are  sensitive  to  this  criticism  and 
believe  they  are  executing  their  responsibilities  as  re¬ 
quired  not  only  by  current  policy  but  also  by  common 
sense.  The  user  and  developer  have  still  different  per¬ 
spectives  on  adequacy  of  testing  and  "how  much  is 
enough?"  Ultimately,  the  Office  of  the  Secretary  of  De¬ 
fense  (OSD)  decisionmakers  must  integrate  the  test  and 
evaluation  (T&E)  data  and  other  issues  which  surround  each 
milestone  review  in  order  to  make  a  rational  decision. 

In  an  overwhelming  number  of  cases,  these  decisions 
have  been  made  conditionally,  i.e.,  they  include  qualifi¬ 
cations  that  often  delay  or  stretch  out  a  program.  This 
study  focuses  on  the  adequacy  of  T&E  data  presented  to 
milestone  reviews  as  a  possible  cause  of  the  conditional 
decision  syndrome.  Reducing  or  eliminating  this  syndrome 
is  one  way  to  shorten  the  acquisition  cycle  and  reduce 
costs;  but,  in  order  to  do  so,  the  underlying  causes  of 
inadequate  T&E  data  or  other  factors  that  result  in  con¬ 
ditional  decisions  must  first  be  understood. 
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No  one  model  exists  which  character i zes  all  asDPctr  of 
the  decisionmaking  process.  This  study  analyzed  individ¬ 
ual  proqram  decisions  in  the  context  of  chanqinq  policy 
and  other  external  events.  However,  there  is  a  wide  vari¬ 
ation  in  the  way  acquisition  policy  is  applied  to  stra¬ 
tegic  as  opposed  to  tactical  or  general  purpose  programs. 
Strategic  and  even  tactical  nuclear  programs  tend  to  oper¬ 
ate  in  an  expedient  fashion  and  are  less  bound  by  estab¬ 
lished  acquisition  policy.  Therefore,  it  may  be  inaooro- 
priate  to  generalize  the  application  of  the  observed 
trends  and  other  findings  of  this  study  to  such  programs. 

This  report  is  organized  in  six  chapters.  Chapters  I 
and  II  provide  background  on  the  relationship  of  T&E  to 
the  DSARC  decisionmaking  process  and  define  the  study  ob¬ 
jectives  and  approach.  Chapter  III  explores  the  wav 
changes  in  acquisition  and  TSE  policy  and  other  external 
events  merge  to  form  a  decision  environment  in  which  the 
milestone  reviews  are  held.  Chapter  IV  investigates  the 
DSARC  decisions  for  the  period  under  studv  as  a  whole  and 
analyzes  trends  and  relationships.  Decision  profiles  of 
six  specific  programs  which  involved  Questions  of  =>d  =  auacy 
of  T&E  data  at  milestone  III  are  provided  in  Chapter  V. 
Finally,  the  findings  of  the  study  are  integrated  into 
conclusions  and  recommendations  which  are  given  in 
Chapter  VI. 


ADDENDUM  TO  FOREWORD 


On  30  April  1981,  during  the  final  publication  o£  this 
report,  Deputy  Secretary  of  Defense  Carlucci  announced  31 
decisions  on  improving  the  acquisition  process,  see  Ex¬ 
hibit  F-l.  These  decisions,  along  with  companion  changes 
to  the  Planning,  Programming  and  Budgeting  System  an¬ 
nounced  on  27  March  1981,  represent  the  Reagan  Administra¬ 
tion's  response  to  oersistent  problems  in  manor  systems 
ac.Tiisi-.  ion  as  perceived  by  Congress  and  the  GAO,  the  OSD 
•  t  j :  *■  ,  the  Services,  and  individual  Program  Managers. 

W :-il«  the  body  of  this  report  has  not  been  revised  to  re- 
f  1  ct  tr.e  new  initiatives*,  several  initial  observations 
-an  no  made  reaarding  the  agreement  between  the  new  poli¬ 
ties  and  procedures  and  the  findinas  of  this  study. 


Cop sea jent ly ,  references  to  "current  policy"  refer  to 
the  March  1980  revision  to  DODD  5000.1  and  DODI 
5000.2,  and  "the  present"  refers  to  the  period  prior 
to  Mav  1931. 


IV 


The  recommendation  of  this  report  that  OSD  seek  added 
funding  flexibility  during  the  critical  transition  from 
R&D  to  production  in  order  to  reduce  the  pressure  on 
making  premature  production  decisions  is  specifically 
addressed  by  the  new  initiatives.  Also,  the  consol ida t ion 
of  OSD  management  attention  at  a  new  milestone  II,  "Pro¬ 
gram  Go-ahead,"  which  authorizes  full-scale  development 
and  production  and  will  occur  somewhat  later  than  the  pre¬ 
vious  miletone  II,  is  consistent  with  the  finding  of  this 
study  that  a  disproportionate  level  of  management  atten¬ 
tion  is  now  focused  on  milestone  III  and  that  a  better 
balance  between  available  information  and  the  ability  of 
the  bureaucracy  to  influence  program  direction  exists 
earlier  in  the  program  at  around  milestone  II.  It  is  not 
yet  clear  how  the  new,  somewhat  later,  milestone  review 
will  be  integrated  into  the  traditional  activities  asso¬ 
ciated  with  full  scale  development  and  it  remains  to  be 
seen  whether  this  will  result  in  substantive  benefits  in 
terms  of  T&E.  It  may  be  that  an  extensive,  delayed  mile¬ 
stone  II  review  will  permit  some  early  developmental  test¬ 
ing  to  be  considered,  but  it  is  also  possible  that  the 
change  in  emphasis  will  alter  the  role  of  operational  test 
and  evaluation  in  the  decisionmaking  process. 

The  new  DSARC  procedure^  also  call  for  the  production 
program  review  (old  milestone  III)  to  be  delegated  to  the 
Service  Secretary.  Consequently,  the  recommendation  of 
this  study  that  the  test  and  evaluation  master  plan  (TEMP) 
be  emphasized  as  a  means  of  achieving  structured  communi¬ 
cation  between  OSD  and  the  Services  is  even  more  important 
than  it  was  previously.  A  clear  understanding  of  the 
measure  of  a  system's  readiness  to  proceed  from  R&D  to 
production  must  exist  in  order  for  OSD  to  successfully 
delegate  the  rate  production  decision  to  the  Services. 

The  TEMP,  if  integrated  into  the  T&E  planning  at  "Require¬ 
ments  Validation"  and  "Program  Go-Ahead,"  will  be  the  pri¬ 
mary  mechanism  for  establishing  and  communicating  these 
requirements.  While  one  can  only  speculate  on  whether 
delegating  the  milestone  III  decision  to  the  services  will 
stand  the  test  of  possible  Congressional,  GAO  or  public 
pressure  for  OSD  intervention,  the  TEMP  could  be  the  in¬ 
strument  which  will  help  maintain  the  controlled  de¬ 
centralization  that  is  now  being  sought. 

Other  new  policy  initiatives,  such  as  providing 
adequate  test  hardware  and  realistically  budgeting  and 
fully  funding  programs,  are  consistent  with  the  findings 
and  conclusions  of  this  study.  However,  the  study  also 
indicates  that  even  increased  funding  and  additional  test 
hardware  will  not  necessarily  be  effective  in  reducing 
conditional  decisions  if  the  critical  test  issues  and  test 
and  evaluation  plans  are  not  identified  and  agreed  to  well 
in  advance  of  the  milestone  review. 
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INITIATIVES  ON  IMPROVING  THE  ACQUISITION  PROCESS 


On  April  30.  1  981.  Deputy  Secretary  of  Defense  Frank  Carlucci  announced  major  changes  both  .n  the  acquisition  pn 
losophy  and  the  acquisition  process  as  practiced  by  the  new  administration  Based  on  a  30  day  assessment  of  the  Qrtrnsr  at 
quisition  system,  the  decisions  address  the  major  problems  in  system  acquisition  perceived  by  Congress  and  the  GAO  he  DSD 
staff,  the  Services  and  Program  Managers  The  major  theme  of  the  changes  is  to  achieve  enhanced  readiness  'educed  *uqu«s 
non  costs  and  shortened  acquisition  time  through  controlled  decentralization  Implementation  of  the  31  decisions  s  present  - 
underway 


1  Management  Principles  include  improved  long-range  planning,  greater  delegation  of  responsibility  authority  and  accouniab>i 
ity.  emphasis  on  low-risk  evolutionary  alternatives,  more  economic  production  rates,  realistic  budgeting  and  full  funding 
improved  readiness  and  sustainability;  and  strengthening  the  industrial  base 

2  Preplanned  Product  Improvement  should  be  used  as  a  means  of  achieving  performance  growth 

3  Multiyear  Procurement  should  be  used,  on  a  case-by-case  basis,  to  reduce  unit  production  costs 

4.  Increased  Program  Stability  in  the  Acquisition  Process  should  be  achieved  by  fully  funding  R&0  and  ptocurement  in  Older 
to  maintain  the  established  baseline  schedule. 

5  Encourage  Capital  Investment  to  Enhance  Productivity  through  legislative,  contractual  and  other  economic  incentives 

6.  Budget  to  Most  likely  Costs  to  achieve  more  realistic  long-term  defense  acquisition  budgets,  reduce  apparent  cost  growth 
and  achieve  increased  program  stability. 

7.  Economic  Production  Rates  should  be  used  whenever  possible  and  advantageous. 

8.  Assure  Appropriate  Contract  Type  in  order  to  balance  program  needs  and  cost  savings  with  realistic  assessment  of  contrac- 
tor  and  Government  risk 

9  Improve  System  Support  and  Readiness  by  establishing  objectives  for  each  development  program  and  "designing-m “  reliabil¬ 
ity  and  readiness  capabilities. 

10  Reduce  the  Administrative  Cost  and  Time  to  Procure  Items  by  raising  the  limit  on  purchase  order  contracts  and  reducing 
unnecessary  paperwork  and  review. 

11-  Incorporate  the  Use  of  Budgeted  Funds  for  Technological  Risk  by  quantifying  risk  and  incorporating  budgeting  techniques 
to  deal  with  uncertainty. 

12.  Provide  Adequate  Front- End  Funding  for  Test  Hardware  m  order  to  emphasize  early  reliability  testing  and  to  permit  concur- 
rent  development  and  operational  testing  when  appropriate 

13.  Governmental  Legislation  Related  to  Acquisition  which  unnecessarily  burden  the  acquisition  or  contracting  process  should 
be  eliminated. 

14  Reduce  the  Number  of  POD  Directives  by  performing  a  cost-benefit  check  and  requiring  that  the  OAE  be  the  sole  issuer  of 
acquisition- related  directives 

15.  Funding  Flexibility  should  be  enhanced  by  obtaining  legislative  authority  to  transfer  individual  weapon  system  procurement 
funds  to  ROTSE  when  appropriate 
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INITIATIVES  ON  IMPROVING  THE  ACQUISITION  PROCESS 

(CONTINUED) 

16.  Contractor  Incentives  to  Improve  Reliability  and  Support  snould  be  developed  and  introduced  into  RFPs.  specifications  and 
contracts 

17  Decrease  DSARC  Briefing  and  Data  Requirements  in  order  to  increase  the  efficiency  of  DSARC  and  other  program  reviews. 

18  Budgeting  Weapons  Systems  for  Inflation  should  be  adopted  in  order  to  more  realistically  portray  program  cost 

19  Forecasting  of  Business  Base  Condition  at  Ma)or  Defense  Plants  by  coordinating  interservice  overhead  data  and  providing 
program  protections  to  plant  representatives 

20  Improve  the  Source  Selection  Process  by  placing  added  emphasis  on  past  performance,  schedule  realism,  facilitization 
plans  and  cost  credibility. 

21  Develop  and  Use  Standard  Operational  and  Support  Systems  to  achieve  earlier  deployment  and  enhanced  supportability  with 
lower  risk  and  cost. 

22  Provide  More  Appropriate  Design  to  Cost  Goals  to  provide  effective  incentives  during  early  production  runs 

23  Assure  Implementation  of  Acquisition  Process  Decisions  by  initiating  an  intensive  implementation  phase. 

24  (ISSUE  A)  DSARC  Decision  Milestones  should  be  reduced  to  Requirements  Validation  '  and  “Program  Go-Ahead."  See 
other  side. 

25.  (ISSUE  B)  MENS  should  be  submitted  with  Service  POM  thus  linking  the  acquisition  and  PPBS  process 

26.  (ISSUE  C)  DSARC  Membership  should  be  revised  to  include  the  appropriate  Service  Secretary  or  Service  Chief 

27  (ISSUE  D)  The  Defense  Acquisition  Executive  (DAE)  should  continue  to  be  the  USDRE 

28.  (ISSUE  E)  The  Criterion  for  DSARC  Review  should  be  increased  to  S200M  RDT4E  and  SIB  procurement  in  FY80  dollars. 

29.  (ISSUE  F)  Integration  0*  the  DSARC  and  PPBS  Process  will  be  achieved  by  requiring  that  fiscally  executable  programs  be 

presented  for  dSARC  review 

30  (ISSUE  G)  Logistics  and  Support  Resources  will  be  included  ;n  the  Service  POM  by  weapon  system,  and  Program  Managers 

will  be  given  more  control  of  support  resources,  funding  and  execution. 

31.  (ISSUE  H)  Improved  Reliability  and  Support  for  expedited  (’  Fast  Track")  programs  will  be  achieved  by  requiring  an  early 

decision  on  the  additional  resources  and  incentives  needed  to  balance  the  risks. 

32.  Increase  Competition  in  acquisition  by  establishing  management  programs  and  setting  objectives.  (July  27.  1981). 
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It  will  undoubtedly  be  some  time  before  the  impact  of 
the  initiatives  on  T&E  policy  and  practice  is  fully  under¬ 
stood.  In  the  meantime,  the  challenge  for  top  management 
is  to  ensure  that  the  new  policies  are  clearly  articulat¬ 
ed,  rapidly  disseminated,  and  effectively  implemented. 

The  challenge  for  individual  program  managers  is  to  adapt 
to  the  current  management  philosophy  while  maintaining  the 
stability  of  existing  program  plans  and  acquisition 
strateg  ies . 
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EXECUTIVE  SUMMARY 


One  explanation  often  given  for  the  propensity  of 
the  DOD  to  equivocate  milestone  decisions  is  that 
inadequate  test  and  evaluation  data  is  presented 
at  milestone  reviews.  This  study  showed  that 
while  most  milestone  II  and  III  decisions  are,  in 
fact,  made  conditionally,  the  adequacy  of  T&E  is 
not  the  predominant  factor.  However,  when  DSARC 
issues  do  arise  which  involve  the  quantity  or 
quality  of  T&E  data  available  for  decisionmaking, 
it  is  generally  because  of  a  lack  of  communica¬ 
tion  between  OSD  and  the  Services.  Clarification 
of  milestone  review  policies,  early  identifica¬ 
tion  of  T&E  issues  associated  with  the  selected 
acquisition  strategy,  and  vigorous  use  of  the 
TEMP  are  some  of  the  ways  this  aspect  of  the  ac¬ 
quisition  process  can  be  improved  in  the  present 
environment  of  decentralized  operations  and  cen¬ 
tralized  decisionmaking. 


The  objective  of  this  study  was  to  gain  a  better 
understanding  of  how  the  adequacy  of  test  and  evaluation 
(T&E)  data  affects  defense  system  acquisition  review  coun¬ 
cil  (DSARC)  decisions.  Specifically,  the  study  focused  on 
the  adequacy  of  the  T&E  data  presented  at  milestone  re¬ 
views  as  a  possible  reason  for  conditional  decisions. 
Conditional  decisions  are  those  that  nominally  approve 
entry  into  the  next  phase  of  acquisition,  but  also  impose 
restrictions  on  the  pace  of  development  or  the  rate  of 
production.  Such  restrictions  invariably  increase  the 
time  and  cost  of  the  acquisition  process. 

Adequacy  of  T&E  data  is  a  difficult  concept  to  define, 
however,  it  is  important  to  distinguish  between  the 
adequacy  of  testing  and  the  test  results  themselves.  In 
this  context,  adequacy  is  defined  as  the  quantity  and 
quality  of  data  that  is  required  for  an  assessment  of  the 
system  as  part  of  an  overall  milestone  review.  Adequacy 
deals  with  whether  the  "right"  tests  were  conducted,  under 
the  "right"  conditions,  in  the  "right"  numbers  rather  than 
whether  the  system  "passed"  or  "failed."  (The  quotation 
marks  indicate  that  this  is  a  simplified  view  of  a  very 
complex  and  frequently  subjective  determination.) 

Numerous  attempts  have  been  made  to  establish  pre¬ 
cisely  the  nature  and  kind  of  testing  required  at  each 
stage  of  the  acquisition  process.  In  general,  these 
attempts  have  not  been  successful  because  adequacy  is  a 
relative  term.  Establishing  the  adequacy  of  T&E  data  on  a 
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given  program  depends  on  the  program's  unique  circum¬ 
stances  and  on  the  disposition  of  the  defense  establish¬ 
ment  toward  caution  or  expediency  as  an  operating  mode. 

In  addition,  as  it  relates  to  the  defense  decisionmaking 
process,  adequacy  of  T&E  data  is  a  strong  function  of  the 
expectations  of  the  decisionmakers  as  well  as  the  environ¬ 
ment  in  which  the  decision  is  being  made.  Consequently, 
for  this  study,  the  DSARC  principals'  own  judgment,  as 
indicated  by  the  decision  documentation,  was  taken  as  evi¬ 
dence  of  the  adequacy  of  T&E  data  for  each  program  studied. 

Nearly  three-quarters  of  the  38  milestone  II  and  III 
reviews  held  between  January  1977  and  August  1980  resulted 
in  conditional  decisions,  and  limited  production,  in  par¬ 
ticular,  was  a  common  means  of  equivocating  the  milestone 
III  decision.  Based  on  the  data  collected,  it  was  found 
that  inadequate  T&E  data  did  contribute  to  some  condi¬ 
tional  decisions,  although  it  was  by  no  means  the  predomi- 
ant  factor.  Breaches  of  performance  thresholds  were  more 
common,  occurring  in  21  of  the  28  conditional  decisions 
made  during  the  period  under  study.  In  addition,  it  was 
found  that  "hidden  agenda"  items  frequently  played  a  role 
in  the  decisionmaking  process.  Nevertheless,  decision 
documentation  on  9  of  the  conditional  decisions  reflected 
a  dissatisfaction  with  the  quantity  or  quality  of  the  T&E 
data  available,  and  it  would  be  desirable  to  reduce  or 
eliminate  such  instances  in  the  future. 

Six  programs  which  appeared  to  involve  inadequate  T&E 
data  at  milestone  III  were  studied  in  detail.  These  in¬ 
cluded  the  Major  Caliber  Lightweight  Gun,  the  Aegis  Weapon 
System,  the  EF-111A  Tactical  Jamming  System,  the  Air 
Launched  Cruise  Missile,  and  the  Patriot  and  Roland  Mis¬ 
sile  Systems.  For  each  system  the  milestone  review  was 
analyzed  in  the  context  of  program-specific  events  as  well 
as  the  overall  decision  environment.  It  was  found  that 
several  of  the  programs  shared  common  attributes,  includ¬ 
ing  that  they  were  longer  than  average,  that  they  had 
attracted  a  great  deal  of  attention  from  the  GAO  and  Con¬ 
gress,  that  in  some  cases  they  were  still  wrestling  with 
fundamental  mission  need  or  concept  selection  issues,  and 
that  there  was  evidence  of  intraservice  disputes  over  pri¬ 
ority  or  the  interpretation  of  the  T&E  results.  Perhaps 
most  importantly,  in  4  of  the  6  cases  an  approved  TEMP  did 
not  exist  early  enough  to  effect  the  adequacy  of  T&E  data 
presented  to  the  DSARC. 

In  the  cases  studied,  inadequate  T&E  data  generally 
resulted  from  a  lack  of  communication  between  OSD  and  the 
Services.  Either  the  expectations  of  the  principals  were 
never  established,  or  there  was  a  change  in  the  program  or 
the  decision  environment  which  was  not  communicated  to  the 


other  party.  Specific  underlying  causes  of  inadequate  T&E 
data  most  frequently  included: 

.  Changes  in  acquisition  policy  since  the  last 

milestone  review  which  altered  the  ground  rules 
for  proceding  from  one  phase  of  the  acquisition 
process  to  the  next. 

.  Shifts  in  emphasis  on  selected  aspects  of  the 

acquisition  process,  such  as  a  preoccupation  with 
reliability  and  maintainability,  by  key  players 
or  groups. 

.  Ambiguities  regarding  the  scope  of  the  system  to 
be  tested,  including  how  much  of  it  would  be 
available  for  OT&E,  how  representative  it  would 
be  of  the  production  configuration  and  how  it 
would  be  evaluated  in  an  operational  environment. 

.  R&D  problems  which  reduced  the  time  and/or  funds 
available  for  T&E. 

.  Problems  in  conducting  the  tests  themselves  due 
to  difficulty  in  obtaining  critical  test  re¬ 
sources  or  to  inherent  range/instrumentation 
limitations. 

Even  when  there  is  evidence  of  inadequate  T&E  data, 
other  factors  tend  to  dominate  the  decision  process.  Par¬ 
ticularly  at  milestone  III,  there  are  enormous  pressures 
to  proceed  with  some  form  of  production  regardless  of  the 
risks  or  the  maturity  of  the  system.  These  pressures  in¬ 
clude  the  national  priority  of  the  program,  the  advancing 
threat,  the  long  lead  times,  and  the  influence  of  an  es¬ 
tablished  constituency  with  specific  vested  interests  in 
the  program's  continuation.  On  the  other  hand,  there  is 
circumstantial  evidence  that  the  adequacy  of  T&E  data 
sometimes  is  used  as  a  convenient  substitute  for  other 
issues  including  near  term  affordability  problems.  In  some 
cases,  it  may  be  easier  for  defense  decisionmakers  to 
stretch  a  program  by  requiring  additional  testing,  thus 
spreading  the  costs  over  time,  then  it  is  to  cancel  any 
given  program  outright. 

Another  conclusion  of  the  study  was  that  formal  commu¬ 
nication  between  OSD  and  the  Services  regarding  the  quan¬ 
tity  and  quality  of  T&E  required  at  each  milestone  deci¬ 
sion  is  critically  important  in  an  environment  which 
employs  decentralized  operations  but  retains  centralized 
decisionmaking.  The  Test  and  Evaluation  Master  Plan 
(TEMP)  has  existed  since  1975  but  was  not  rigorously  used 
until  about  1979.  Although  the  TEMP  is  now  viewed  by  some 
as  a  "contract"  between  the  parties,  its  real  value 
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appears  to  be  as  a  mechanism  for  insuring  continuing 
structured  communication  regarding  the  T&E  program.  How¬ 
ever,  it  is  important  that  T&E  plans  as  shown  in  the  TEMP 
accurately  reflect  the  program's  response  to  any  c.nanges 
in  the  fundamental  assumptions  to  the  acquistion  strategy. 

It  was  also  found  as  part  of  this  study  that  improve¬ 
ments  are  needed  in  the  manner  in  which  T&E  data  is  inte¬ 
grated  into  the  milestone  review  process.  First,  the 
independent  T&E  assessment  by  the  DDT&E  is  an  important 
and  valuable  input  to  DSARC  decisionmaking;  however,  it  is 
but  one  part  of  an  overall  evaluation  that  is  not  pres¬ 
ently  accomplished  in  a  rigorous  fashion.  Such  an  overall 
evaluation  must  view  the  existing  T&E  data  in  the  context 
of  a  formal  risk  assessment  of  the  alternative  courses  of 
action.  Second,  it  was  found  that  many  of  the  questions 
regarding  adequacy  of  T&E  data  at  milestone  III  reviews 
arise  out  of  the  immaturity  of  the  hardware  or  software 
used  for  DT&E  and  IOT&E.  Although  present  policy  author¬ 
izes  initial  production  for  IOT&E  along  with  full  scale 
development  at  milestone  II,  there  appears  to  be  a  number 
of  misconceptions  and  barriers  which  have  prevented  full 
implementation  of  this  approach.  Third,  the  study  showed 
that  a  disproportionate  amount  of  management  attention  is 
now  focused  on  milestone  III,  by  which  time  there  are  few 
realistic  alternatives  to  proceeding  with  some  form  of 
production.  Although  there  are  explanations  for  this  phe¬ 
nomenon,  it  was  found  that  a  better  balance  between  avail¬ 
able  information  and  the  ability  of  the  bureaucracy  to 
influence  program  direction  occurs  earlier  in  the  program 
about  the  time  of  milestone  II.  In  this  case,  the  empha¬ 
sis  of  the  milestone  review  necessarily  must  be  on  the 
identification  of  critical  test  issues  and  on  the  proposed 
T&E  plans  rather  than  on  extensive  T&E  results.  Although 
the  study  has  shown  that  lack  of  a  common  understanding  of 
T&E  issues  and  plans  is  a  fundamental  cause  of  subsequent 
problems,  such  a  shift  in  emphasis  raises  questions  re¬ 
garding  the  role  IOT&E  in  the  decisionmaking  process. 

Finally,  it  was  found  that  there  are  institutional 
pressures  which  pose  obstacles  to  the  developer  and  tester 
in  conducting  an  adequate  T&E  program.  Competition  for 
resources  and  other  interservice-related  factors  tend  to 
cause  success  oriented  planning  without  adequate  flexibil¬ 
ity  to  deal  with  inevitable  R&D  problems.  As  milestone 
review  dates  approach,  developers  are  forced  to  limit 
testing  in  order  to  solve  other  problems.  Frequently, 
premature  production  reviews  were  held  because  of  a  per¬ 
ception  that  milestone  III  approval  was  necessary  in  order 
to  preserve  follow-on  production  funding.  Other  institu¬ 
tional  id iosyncracies  include  the  fact  that  changes  in 
program  funding  due  to  perturbations  in  the  PPRS  process 
are  seldom  coupled  to  official  changes  to  the  acquisition 
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strategy  and  that  the  allocation  of  test  range  costs  under 
the  uniform  funding  policy  sometimes  acts  as  a  disincen¬ 
tive  to  conduct  comprehensive,  thorough  testing. 

As  shown  in  Exhibit  VI-1,  11  recommendations  were  pro¬ 
posed  in  response  to  the  findings  and  conclusions  of  the 
study.  These  are  listed  below  and  discussed  in  detail  in 
Chapter  VI. 

.  The  DSARC  should  place  greater  emphasis  on  ear¬ 
lier  milestone  reviews  in  order  to  influence  and 
optimize  the  acquisition  strategy  without  major 
economic  or  schedule  disruptions. 

.  OSD  should  clarify  the  policy  regarding  the  mean¬ 
ing  of  milestones  II  &  III  so  that  initial  pro¬ 
duction  hardware  is  routinely  evaluated  during 
IOT&E . 

.  New  resource  programming  techniques  should  be 

devised  which  foster  the  transition  from  R&D  to 
production  and  prevent  premature  production 
reviews . 

.  OSD  should  obtain  better  consensus  regarding  the 
meaning  of  "Operational  Environment"  for  each 
major  system  so  that  future  confusion  is  avoided. 

.  More  rigorous  T&E  planning,  including  the  use  of 
system  engineering  discipline,  should  be  used  to 
define  the  scope  of  the  system  to  be  tested  and 
avoid  ambiguities  which  lead  to  differing  expec¬ 
tations  by  defense  decisionmakers. 

.  OSD  snould  reiterate  and  emphasize  use  of  the 

TEMP  as  the  principal  means  of  communicating  the 
T&E  requirements  for  proceeding  from  one  phase  of 
acquisition  to  the  next. 

Better  overall  evaluations  should  be  conducted  as 
part  of  the  milestone  review  process  including 
more  rigorous  analysis  of  alternative  courses  of 
act  ion . 

.  Program  Managers  should  submit  impact  statements 
on  their  programs  following  changes  in  acquisi¬ 
tion  policy  so  that  compliance  issues  can  be  sur¬ 
faced  as  early  as  possible.  Also,  OSD  should  be 
more  sensitive  to  documenting  changes  in  emphasis 
on  the  acquisition  process  as  it  occurs. 

.  The  DDT&E  should  submit  recommendations  regarding 
institutional  improvements  in  T&E  after  each 
DSARC  in  whicn  T&E  problems  are  an  issue. 


OSD  should  continue  to  monitor  the  relationship 
between  T&E  data  and  DSARC  decisions  in  order  to 
determine  whether  present  policies  regarding  the 
TEMP  are  effective. 

Research  should  be  undertaken  to  determine  the 
underlying  causes  of  performance  breaches  as  the 
predominant  contribution  to  conditional  decisions. 
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I NTRODUC  I'lON 


This  r^Dor:  desc  r  i a  r---r.->.ir;'h  st  nr-  ■  •  :  : 

3ooz,  Alien  &  Hamilton  Inc.,  for  the  Dof  of>2<-  Syst 
uoeino.nt  College,  Ft.  1  vo  :  r  ,  '/ini.nia.  !’:•.>•  v, 
the  study  was  to  gain  2  better  understanding  of  now  t;. 
adequacy  of  test  and  •'valuation  (TSE)  iata  2  f :  -'-c  t  a 
System  Acquisition  Review  Council  (DSARC)  decisions, 
study  addressed  the  Id  mii-eston-1  II  and  III  r«v  i--w: 
net ween  January  1977  and.  August  1980. 


The  role  of  r&E  in  the  d»»c is iomnak i.nu  process  has 
changed  profoundly  over  the  last  several  decades,  and. 
today  is  not  universally  understood.  The  Department 
Defense  (DOD)  has  tried  repeatedly  to  find  wavs  to  sh 
the  acquisition  cycle  without  incurring  maccepcar. le 
risKS.  But  the  risks  are  elusive  without  adequate  te 
and  the  adequacy  of  T&E  data  is  a  dif f  icult  concept  t 
define.  However,  in  defining  the  concept ,  it  can  be 
stated  tr.at  "adequate"  test  me  and  iata  is: 


A  strong  function  of  the  expectations  of  the 
dec  1 s ionma  ke  r  s 


Related  to  whether  test  ina  is  comprehens i ve ,  r 
orous,  and  complete 

Different  from  the  assessment  of  performance 
Only  one  part  of  the  overall  evaluation  proces 


The  DSARC  principals  may  qualify  their  decisions 
thus  restrict  the  pace  of  development  and  production 
cause  of  inadequate  T&E  data.  It  is  also  possible, 
ever,  that  the  adequacy  of  T&E  'iata  is  a  convenient 
at  mate  for  other  issues  that  are  more  difficult  to 
w  i  t  h . 


Thi;  cnapter  gives  a  brief  oac*o round  of  tee  oro.ole 
under  study,  describes  the  relationship  of  T&E  data  to 
DSARC  process,  enumerates  study  objectives,  and  review? 
previous  analyses. 

1 .  BACKGROUND 

In  the  decade  since  former  Secretary  of  Defonso 
Melvin  R.  Laird  and  Deputy  Secretary  of  Defense  David 
Packard  began  major  reforms  and  improvements  in  the 
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ietenne  system;;  acquis  it  ion  or  ickcs,  »•  he  r  ••  has  h>***r:  : 
litany  v  proposal  approaches  t  v ;  no 

lie,  and  performance  peon  loros  which  oUwjf-i  new  we  a  cor. 
systems  then  as  they  do  today.  Among  the  m  a  nan  •’".<- ot  t:v-o 
ries  and  styles,  the  procedures,  and  documentation,  two 
t  semes  em-'-r  ;  o .  The1  first  is  tnat  i  sort  of  ivnurnic  t".n- 
3 ion  has  always  existed  between  those  manaaer s  who  woul  j 
accelerate  system  developments  at  ail  costs  (and,  know¬ 
ingly  or  not,  accept  the  risks  therein)  and  those  who 
would  procee-a  in  a  more  cautious,  step-by-step  fashion. 
The  latter  group,  while  certainly  not  opposed  to  acceler¬ 
ation  the  acquisition  process,  and  not  unmindf j!  of  the 
costs  associated  with  long  development  times,  s i mol v  tax*? 
a  oroader  view  of  the  ultimate  cost  of  acquiring  a  weapon 
system,  i.e.,  the  cost  to  do  it  right  versus  the  cost  to 
do  it  over. 

Not  everyone  agrees  that  these  oosi  t  ions  are  mut  J.allv 
exclusive;  nowever,  most  observers  of  the  acquisition  pro 
cess  agree  on  the  second  theme:  that  the  mood  of  th«=  de¬ 
fense  establ  i  shment  invariably  swims  like  a  pendulum,  to¬ 
ward  one  extreme  and  then  the  other.  Dunne  tne  IhhOs, 
for  example,  concurrency  of  development  and  production  wa 
an  accepted  technique  for  reducing  development  time.  Du r 
mg  the  1970s,  it  was  officially  proscrioed  due  to  numer¬ 
ous  technical  failures  which  were  experienced.  Now ,  with 
the  March  1980  publication  of  DOD  Directive  5000.1  and  DO 
Instruction  5000.2  ,  the  technique  is  cautiously  bom:  re¬ 
introduced.  Other  examples  of  snifts  in  management  phi¬ 
losophies  include  centralized  and  decentralized  decision- 
making,  total  package  procurement  and  "fly-before-buy," 
government  direction  of  contractor  efforts,  and  hands-off 
strategies,  etc.  At  one  point,  the  initial  operational 
capability  (IOC)  was  considered  a  parameter  subject  to 
tradeoffs  just  like  performance  and  cost.  Now,  acceler¬ 
ated  deployment  of  systems  seems  to  be  the  dominant  theme 

TSE  is  the  pivot  on  which  this  pendulum  swines.  Seme 
see  TSrE  as  the  only  means  of  objectively  measuring  the 
anility  of  a  system  to  meet  an  expressed  need  and  as  the 
primary  vehicle  for  deciding  when  to  make  proa r ess i ve ly 
larger  investments  in  response  to  diminishina  risks. 
Others  see  T&E  principally  as  feedback  to  the  design  pro¬ 
cess  and,  except  in  extreme  cases,  laraely  unrelated  to 
the  decision  to  proceed  to  the  next  phase  of  accruisition. 
This  view  of  TSE  is  caused  by  a  perception  that  it  Gener¬ 
ally  would  be  too  costly  and  time  consuming  to  do  other¬ 
wise.  The  pressure  on  the  one  hand  is  to  no  faster  and, 
on  the  other  hand,  to  avoid  makina  a  mistake. 
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The  DSARC  is  the  DOD  body  charged  with  advising  the 
Secretary  or  Dp  r  tn.:;.' ,  op.  a  casc-ov-case  oasis,  w::;c:  of 
these  two  paths  is  'nost  desirable.  While  Historically  the 
numoer  ot  >utr  iqht  c  i.nee  1  i  at  ions  recommend*-.’  by  the  DSARC 
has  seen  small,  there  has  been  a  trend  in  recent  years  to 
equivocat-'  tne  iecision  to  proceed,  i  .  ,  to  stretch  out 

programs  and  proceed  at  a  slower  pace  while  oroolems  are 
resolved,  or  to  attach  qualifications,  often  involving 
additional  test  and  evaluation,  to  the  systems'  continued 
ievelopaient  or  production.  Some  view  such  "T&E  strings" 
simply  as  micromanagement  by  OSD;  others  see  them  as  sub¬ 
stitutes  for  more  difficult  issues.  In  either  case,  this 
situation  invariably  lengthens  the  acquisition  cycle. 

In  the  38  milestone  II  and  III  decisions  made  netween 
January  1977  an  1  August  1980,  nearly  three-quarters  im¬ 
posed  conditions  of  one  sort  or  another.  The  sc.nool  of 
thought  that  would  favor  faster  development  would  take 
strong  exception  to  trie  additional  development  costs,  un¬ 
economical  procurement  buys,  and  the  stretched  out  TOCs 
that  resulted.  However,  tne  school  favoring  demonstrated 
performance  would  be  frustrated  by  the  frequency  that  the 
Services  brought  unsatisfactory  items  to  the  DSARC  for 
review.  Both  schools  can  only  be  dismayed  by  the  ineffi¬ 
ciency  of  the  process  and  hope  that  it  can  be  improved. 

2.  TSE  DATA  AND  THE  DSARC 

The  defense  establishment  instituted  a  formal  mile¬ 
stone  review  process  more  than  10  years  ago  and  has  con¬ 
tinuously  refined  it.  Yet,  DSARC  reviews  continue  to 
reveal  deviations  from  plans,  performance  failures, 
affordability  issues,  and  other  unexpected  occurrences. 

One  explanation  might  be  that  if  everything  passed  DSARC 
review,  then  DOD  simply  was  not  being  as  aggressive  or 
ambitious  as  it  should  he  in  its  selection  of  projects. 
After  all,  some  failures  must  be  expected  ic  doctrinal  and 
technical  barriers  are  to  be  overcome.  However,  a  more 
realistic  view  in  light  of  the  severe  budget  constraints 
facing  DOD  is  that  there  are  too  many  deviations  from 
agreed-to  plans  and  that  "disconnects"  are  occurrina 
between  the  Services  who  are  developing  the  systems,  and 
the  OSD  who  is  reviewing  them. 

DSARC  decisionmaking  must  be  viewed  as  an  amorphous 
process  which  is  affected  as  much  by  the  predisposition 
and  personalities  of  the  principals,  and  by  exogenous 
events  and  the  tenor  of  the  times,  as  it  is  by  the  iso¬ 
lated  facts  regarding  the  sytem  under  review.  Neverthe¬ 
less,  since  the  later  1970s,  there  has  been  increasina 
emphasis  on  supporting  milestone  decisions  with  hard  data 
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3.  STUDY  OBJECTIVES 


Th  i  a  study  focused  on  tne  issue  of  adr-au  acv  r'  T.&E 
iota  as  it  relates  to  the  decisionmakma  process.  In  par¬ 
ticular,  the  study  attenuated  to  determine  wnetr.er  inade¬ 
quate  T&E  data  result  m  conditional  decisions  and,  if  so, 
what  were  the  underlying  causes  of  inadequate  data.  Re¬ 
search  conducted  during  the  course  of  the  study  also  ex¬ 
plored  other  possibilities  for  exol  a  i  n  i  na  trie  conditional 
DSARC  decisions  and  the  way  adequacy  of  testina  is  treated 
in  the  milestone  review  process. 

The  following  specific  questions  guided  toe  research 
efforts: 

.  Are  conditional  decisions  caused  hv  inadecuate 
TSE  data?  Or  are  there  other  explanations? 

.  If  conditional  decisions  are  caused  by  inadequate 
data,  what  are  the  under Iv i na  causes  of  suer, 
situations? 

.  Is  there  effective  communication  between  OSD  and 
the  Services  with  respect  to  the  quantity  and 
quality  of  T&E  data  required  for  milestone 
dec i s  ions? 

.  Is  the  adequev  of  T&E  data  effectivelv  addressed 
in  the  present  milestone  review  process? 

.  What  are  the  significant  obstacles  faemo  the 

developer  and  tester  in  planning  and  conductina 
an  adequate  T&E  program? 

4.  PREVIOUS  ANALYSIS 
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In  1972,  the  General  Accountina  Office  (GAO)  issued 
its  first  major  reoort  cn  T&E  problems  in  the  DOD.  En¬ 
titled  The  Imoortance  of  T&E  in  the  Acau is i t  ion  Process 
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f  t  '!J~or  '.'.'P3i)on  Systems,  the  report  cone:  idol  tr.  : 
systems  studied  .lid  not  nave  adequate  tost  olans  i 
r&E  wi  a  not  accomplished  in  a  timely  manner.  Most 
tantiv,  tne  GAO  also  concluded  that  T&E  data  we r o 
ava:  la  ole  prior  to  those  times  in  tne  acquisition 
when  decisions  had  to  be  made.  This  report  may  ha 
tr  muted  to  tne  "T&E  explosion"  w:i  icn  occurred  dur 
1970s.  However,  tne  reform  in  TSE  policy  and  the 
creased  empnasi  s  on  T&E  as  a  means  of  improving  t:i 
sition  process  actually  began  with  the  President's 
Rincon  Defense  Panel  in  July  1970  and  the  Deputy  S 
of  Defense  memos  on  TSE  to  the  Service  Secretaries 
February  and  August  1971. 
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The  Blue  Ritbon  Defense  Panel  was  one  of  tne  first 
groups  to  articulate  problems  with  Total  Package  Proem 
ment  polciies  tnat  were  then  in  effect.  The  oane 1 
stressed  the  importance  of  periodic  review  of  acquis  it ' 
programs  at  critical  milestones.  These  reviews  would 
vide  an  opportunity  to  decide  to  continue,  reorient,  o: 
cancel  a  program  based  on  a  comoarlson  of  that  program 
progress  to  previously  established  goals  and  objective: 
The  impact  on  test  and  evaluation  was  obvious.  In  ori< 
to  be  effective,  decisionmakers  needed  relia.ole  and  ac¬ 
curate  measurements  of  program  progress  compared  to  pr- 
gram  objectives. 
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The  coming  empnasis  on  T&E  was  anticipated  =,ornwr.a 
Deputy  Secretary  Packard.  In  May  1969,  he  established 
DSARC  as  the  formal  DOD  acquisition,  program  review  ooc 
and,  in  mid-1971,  he  issued  a  series  of  memoranda  wnic 
established  a  focal  point  for  T&E  within  the  Deputy  Di 
tor.  Research  and  Engineering  ( DDR&E )  .  lie  also  repair 
eacn  Service  to  establish  independent  test  agencies  an 
strong  headquarters  staff  focal  points,  3nd  generally 
vated  the  importance  of  T&E  in  the  acquisition  process 


Since  then,  there  has  been  continued  int°r-->-t  in 
revision  of  DOD  T&E  policy.  Numerous  groups  and  :  *:d  iv:  -- 
uals,  both  internal  and  external  to  Government,  have 
searched  for  solutions  to  w.nat  one  researcher  called  "Tne 
Persistent  Problem."  The  Defense  Science  Board  (DS31  es¬ 
tablished  a  Task  Force  on  Test  and  Evaluation  in  l3'-} 
which  developed  guidance  on  test  and  evaluation  through 
examination  of  general  issues  and  tne  generation  of  -'heck 
lists  oriented  toward  good  procedures  and  practices 
relative  to  T&E.  At  the  same  time,  the  Deputy  Director 
(Test  and  Evaluation)  published  a  series  of  reports 
establishing  T&E  guidelines  for  various  types  of  weapon 
systems.  In  1977,  the  DSB  formed  a  Task  Force  on  Test  an 
Evaluation  Policies  which  explored  broader  questions 
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regard  mu  t  ho  role  of  T&E  in  the  acquisition  process. 
However,  these  and  other  studies  have  not  focused  on 
inadequate  T&E  fata  as  a  possible  explanation  for 
conditional  program  lec  isions.  A  selected  annotated 
bibliography  of  related  studies  is  contained  in  Appendix 
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II. 


STUDY  APPROACH 


DS ARC  decisions  represent  a  unique  blerd  of  technical, 
economic,  and  political  factors.  Consequentl'-' ,  the  tech¬ 
nical  approach  used  for  this  study  was  to  categorize  and 
define  the  apparent  reasons  for  conditional  decisions  and 
the  underlying  causes  of  inadequate  T&E  data  based  on  pro¬ 
gram  unique  issues  and  the  overall  environment  in  defense 
proaram  decisionmaking  for  the  period  of  interest.  As 
shown  in  Exhibit  II-l,  the  study  progressed  in  five  steps: 

Define  the  study  objectives 

.  Identify  and  characterize  DSARC  decisions  for  the 
period  January  1977-August  1980 

.  Describe  the  decision  environment  with  respect  to 
policy  and  external  events 

.  Select  the  subset  of  decisions  which  seem  to  in¬ 
volve  inadequate  T&E  data  for  in-depth  study  in 
the  context  of  the  decision  environment;  identify 
common  elements,  underlying  causes  and  other  pos¬ 
sible  reasons  for  the  decision  and  its 
consequences 

.  Integrate  the  findings  and  draw  conclusions  and 
make  recommendations  for  improving  the  situation 
in  the  future. 

This  chapter  presents  some  definitions  and  establishes 
a  framework  for  discussing  the  DSARC  decision  process. 

Data  collection  and  analysis  techniques  that  were  used  are 
also  described. 

1.  DEFINITIONS 


The  primary  goals  of  DOD  T&E  activities  are  the  aath- 
ering  and  analysis  of  test  data  and  the  formulation  of 
assessments  that  support  the  acquisition  decision  pro¬ 
cess.*  Identification  of  critical  issues  with  measurable 
goals  and  thresholds  and  statistically  significant  sample 
sizes  is  a  worthy,  if  sometimes  impractical,  objective. 
However,  when  asked  to  define  adequacy  of  T&E  data,  one 


Introduction  to  the  Application  of  Statistical  Con¬ 
cepts  to  Test  and  Evaluation.  Office  of  the  Director, 
Defense  Test  and  Evaluation,  August  1980. 
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EXHIBIT  1 1 - 1 
Technical  Approacn 


OSD  official  commenced  that  the  DSARC  was  looking  for 
"...reasonable  evidence  of  a  reasonable  amount  of  testing 
that  was  reasonably  successful."  While  imprecise,  this 
definition  is  probably  very  nearly  an  accurate  description 
of  what  actually  happens  when  the  DSARC  principals  try  to 
integrate  the  hard  and  soft  data  presented  to  them. 

Numerous  attempts  have  been  made  to  define  in  absolute 
terms  the  amount  and  kind  of  test  and  evaluation  required 
during  each  phase  of  the  acquisition  cycle.  For  example, 
the  DDT&E  guidelines  for  various  types  of  weapon  systems 
provide  excellent  checklists  of  things  that  should  be  con¬ 
sidered.  However,  financial,  temporal  and  other  pressures 
almost  always  restrict  the  amount  of  testing  that  is  re¬ 
alistically  possible  and  a  generally  accepted  formula  has 
never  been  devised  for  conducting  trade-offs  between  test¬ 
ing  and  other  program  activities.  Thus,  in  the  final 
analysis,  the  determination  of  adequacy  of  testing  in  a 
given  case  rests  with  the  decision  makers  themselves  who 
must  weigh  the  costs  and  risks  of  various  alternatives. 
Therefore,  for  this  study,  the  DSARC  principals'  own 
statements  are  taken  as  evidence  of  the  adequacy  of  T&E 
data,  i.e.,  the  adequacy  of  T&E  data  is  defined  as: 

The  quantity  and  quality  of  data  which  in  the  judge¬ 
ment  of  the  DSARC  principals  are  necessary  for  an  as¬ 
sessment  of  the  system  in  the  context  of  an  overall 
evaluation . 

Note  that  this  definition  does  not  address  the  inherent 
information  content  of  the  data.  That  is,  the  adequacy  of 
the  testing  is  differentiated  from  the  test  results  them¬ 
selves.  Th  s  point  is  shown  graphically  in  Exhibit  1 1  - 2 . 

Viewed  simplistically ,  tests  can  be  divided  into  two 
classes,  right  and  wrong,  where  wrong  tests  include  those 
that  are  not  compr ehens i ve ,  rigorous,  or  complete.  The 
outcome  of  a  test  can  be  broadly  classified  as  either  pass 
or  fail,  depending  on  the  evaluation  criteria,  the  in¬ 
herent  quality  of  the  system  design,  and  the  interpreta¬ 
tion  of  the  results.  This  leads  to  a  two-by-two  matrix  of 
possible  events  : 

.  The  right  tests  are  conducted  and  the  system 
passes,  in  which  case,  all  other  things  being 
equal,  the  decision  is  rather  st ra ightf orwa rd  and 
an  unconditional  decisions  is  made  to  proceed  to 
the  next  phase. 
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'WRONG"  MEANS  THOSE  TESTS  THAT 
ARE  NOT  COMPREHENSIVE.  RIGOROUS 
OR  COMPLETE 


.  The  riqht  tests  are  conducted,  hut  tne  system 

fails  ail  or  some  portion  of  them.  This  tn<- 
arena  of  management  j udaement  in  wnicn  tne  risks 
of  or  aoeedina ,  in  the  hopes  of  max  i nq  the  system 
work,  are  weighted  against  the  risks  associated 
with,  delay  inq ,  fixing,  and  retest  inq.  This  is 
the  classic  case  in  which  defense  leaders  rou¬ 
tinely  disagree  because  of  differences  in  the  i  r 
own  perspectives  and  attitudes. 

The  system  apparently  passes,  nut  the  tests  are 
judcied  by  the  DSARC  to  be  the  wrong  ones.  Hero 
the  situation  is  not  so  clear,  even  with  regard 
to  tne  structure  for  debating  alternative  courses 
of  action.  While  it  is  possible  to  define  the 
risks  in  not  proceedina  (increased  cost,  ad¬ 
vancing  threat,  etc.),  depending  on  how  wrong  the 
tests  are,  there  are  inadequate  data  to  ration¬ 
ally  assess  the  risks  in  proceeding.  For  exam¬ 
ple,  it  might  be  difficult  to  assess  the  r  1  sk  s  in 
proceeding  with  a  system  that  is  required  to  op¬ 
erate  in  an  electronic  countermeasures  (ECM)  en¬ 
vironment  if  it  had  not  been  tested  under  ECM 
cond itions. 

.  The  last  case,  where  the  system  fails  the  wror.a 
tests,  is  nearly  identical  to  the  preceding. 
Deoendinq  on  whether  one  was  an  advocate  or  an 
antagonist,  it  would  be  easy  to  be  critical  of  a 
system  with  bad  test  scores.  Unfortunately,  if 
the  tests  were  somehow  inappropriate  or  imprt.Der, 
this  miaht  be  overly  harsh  and  result  in  hoidina 
a  svstem  back  unnecessarily.  For  example,  test¬ 
ing  the  ability  of  a  sinale  aircraft  to  penetra 
enemy  air  defenses  when  the  intended  tactic  is 
penetrate  in  mass  or  in  conjunction  with  other 
systems  might  make  difficult  a  rational  decision 
to  proceed.  [It  must  be  pointed  out,  however, 
that  these  data  might  be  very  significant  in  an 
overall  evaluation  because  it  would  pound  the 
expected  performance  of  the  system.] 

The  last  two  cases  involve  inadequate  T&E  data;  i.e., 
the  tests  themselves  are  determined  by  the  DSARC  princi¬ 
pals  to  be  somehow  lacking  and  the  resulting  data  are 
judged  to  be  inadequate  to  make  an  uncond i t iona 1  decision 
to  proceed. 
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The  adequacy  of  r&E  data  involves  both  quantity  and 
.lu.i  1 :  tv .  Quantity  deals  with  tne  probability  of  success, 
confidence  limits,  and  the  incremental  worth  of  collection 
uid i t 1 ona l  d.ta  versus  the  risk  of  makinq  an  incorrect 
decision.  Quality  issues  deal  with  the  realism  of  the 
test  ' o . q . ,  testing  Copperhead  under  realistic  battlefield 
conditions  such  as  smoke  and  foq)  or  the  lack  of  specific 
t ecnnical  or  operational  tests  thought  to  be  important 
e.q.,  testing  the  F-16  at  the  extremes  of  its  oerformance 
envelope  ar  testinq  Patriot's  compatioil ity  with  related 
air  iefense  and  command  and  control  equipment). 

d .  FRAMEWORK  FOR  DSARC  DECISIONS 

Exhibit  T I  —  3  shows  a  model  of  the  DSARC  decision  nro- 
cess  in  which  various  types  of  assessments  and  the  overall 
evaluation  taKe  place  in  and  react  with  the  decision  envi¬ 
ronment.  The  decision  environment  includes  the  estab¬ 
lished  policies,  the  attitudes  of  defense  decisionmakers, 
and  other  external  events,  such  as  Cona res s iona 1  involve¬ 
ment  or  decisions  on  other  related  programs.  Any  of  the 
assessments  can  lead  to  conditional  decisions.  For  exam¬ 
ple,  a  judgement  that  there  is  inadequate  T&E  may  result 
in  delays  in  order  to  conduct  add itional  tests;  a  perfor¬ 
mance  assessment  which  indicates  a  failure  to  meet  a  par¬ 
ticular  threshold  may  result  in  a  fix  and  retest  cycle; 
and  other  assessments  may  result  in  slowing  tne  pace  of 
the  program  for  other  specific  reasons. 

Within  this  framework  3  possible  syntax  for  discussion 
DSARC  decisions  is  shown  below: 


SUPPORTED  DUE 


DECISION 

BY 

APPARENT 

TO 

UNDERLYING 

REASONIS) 

CAUSES 

1 


The  syntax  proposed  is  that  various  decisions  a  re  caused 
by  one  or  more  reasons  which,  in  turn,  are  due  to  under  Iv¬ 
ina  causes.  The  decision  itself  can  be  of  several  tvpes, 
such  as  unconditional  approval  to  proceed  to  the  next 
phase,  varying  deqr°es  of  conditional  approval,  or  denial 
or  delay  in  proceeding  including  proa  ram  cancellation. 

For  each  decision  tvoe,  there  are  a  number  of  apparent 
reasons  generally  given  by  th°  DSARC  princioals  as 
justif ication. 


1 
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exhibit  rr-3 
DSAPC  Decision  Model 
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Die  sec  of  aoparent  reasons  can  :  *?  let'  me-.i  bv  1  ~>ok  :  no 
at  both  the  management  and  system  issues  tnat  are  reviewed 
at  milestones  TI  and  III  t  Fx.o  ibi  t  T  I  — i )  .  1 1  be  seen 

tnat  the  issues  q  roup  into  tour  general  areas:  need ,  af- 
t  or  i  no  1 1 1  ty  ,  *  maturity,  and  management  'o!  nnninq.  Fabar- 
to  address  each  of  these  issues  satisfactorily  can  const  i  - 
Cute  a  reason  for  less  than  unconditional  aiacrovai. 

These  issues  apoear  loqical  in  terms  of  a  rational 
dialogue  concerning  r  isK  and  investment  of  additional  re¬ 
sources.  In  order  of  importance,  they  address  the  follow¬ 
ing  questions : 

.  Does  the  system  address  a  legitimate  need? 

.  Is  the  system  affordable? 

.  Is  it  sufficiently  mature  and  -...'ill  it  likelv  meet 
its  technical  and  operational  goals? 

.  Are  the  program  manaaement  and  planr.ina  asoects 
of  the  program  well  in  hand? 

While  T&E  touches  on  all  four  categories,  it  is  most  fre¬ 
quently  raised  as  an  issue  relatinq  to  maturity  and,  less 
often,  to  management  and  planning. 

If  the  apparent  reasons  for  a  particular  decision  have 
been  determined,  then  an  attempt  can  be  made  to  identify 
and  understand  the  underlying  causes.  For  examole,  one 
possible  underlying  cause  of  a  failure  to  oresent  adequate 
T&E  data  to  the  DSARC  fa  maturity  failure)  is  ooor  orepa- 
r at  ion  or  coordination  of  the  test  and  evaluation  master 
plan  (TEMP) .  Another  is  the  lack  for  availability)  of 
appropriate  test  facilities  or  instrumentation.  A  thir-' 
miqht  be  insufficient  time  or  money  to  conduct  a  compre¬ 
hensive  test  program. 


There  are  many  definitions  of  affordability ;  however, 
the  one  adopted  here  is  simnly  the  ability  to  provide 
adequate  resources  to  acquire  and  operate  a  system  in 
an  efficient  and  effective  manner.  The  "bow  wave" 
(discussed  later)  refers  to  the  financial  c*>nd  i  -  ion  i 
which  more  systems  are  developed  than  can  he  procured 
and  deployed  within  ant ic ioated  defense  budgets. 
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OSARC  II 


DSAHC  lit 


NEED 


AFFORD 

ABILITY 


•  MISSION  ELEMENT  NEED  TASK  REAFFIRMED  AND  •  MISSION  ELEMENT  NEEO  REAFFIRMED  ANO  INTERACTIVE 

INTERACTIVE  THREAT  ANALYSIS  UPDATED  THREAT  ANALYSIS  UPOATEO 

•  SYSTEM  SELECTED  MEETS  NEED  AND  iS  COST  E  FFECTI VE 


•  AFFORDABILITY  CONF  IRMEO  LIFECYCLE  COSTS  WITHIN 
-^MOUNTS  REFLECTED  IN  LATEST  F  YQP'EPA 


-  aC  HE  DOLE  ANO  COST  FSTiMATES  MfAi  1ST  !L 
•  I  •  •  »  (.  l  E  COSTS  WITHIN  AMOUNTS  RE  I  i  i  L  H  .>  IN 


•  PROGRAM  AND  FISCAL  YEAR  THRESHOLDS  REAFFIRMED 


MATURITY 


MANAGE 
MENT  ISSUES 
ANO  PLANS 


•  TRADE  OFFS  BFn*F  EN  COST  PERFORMANCE  SCHEDULE 
AND  LOGIST'C  SUPPORT  ABILITY  ARE  ACCEPTABLE 

•  UNOF  *•  TAIN  TIES/ RISKS  ARE  IDENTIFIED  ANO  ACCEPTABLE 

•  SYSTEM  REQUIREMENTS  AOEQUATEL  V  SPECIFIED  AND 
INCLUDE 

PERFORMANCE 

OESIGN  TO-COST  ANO  LIFE  CYCL.  COST 

NUCLEAR  SUHVIVA8IUTY 

ELECTROMAGNETIC  COMPATIBILITY 

E  l  EC  TRONIC  NF  RARED  OPTICA  4.  COUNT  E  HME  AbUHE  S 

PRODUCBILITY 

-  NATO  STANDARDIZATION  ANO  iNTE  R  OPF  R  ABIL  I  T  Y 
LOGISTICS  SUPPORT  ABILITY 

•  RELIABILITY  AND  MAINTAINABILITY  GOALS  AND 
THRESHOLDS  ESTABLISHED 

•  MANPOWER  GOALS  ANO  THRESHOLDS  IDENTIFIED 


•  UPDATED  ACQUISITION  STRATEGY  SUPPORTS  PROGRAM 
OBJECTIVES  AND  'S  BEING  PROPERLY  EXECUTED 

•  PROGRAM  MANAGEMENT  STRUCTURE  SOUND  ANO 
ADEQUATELY  SUPPORTED 

•  PROBLEMS  MANAGEMENT  PLANS  COMPLE  TE  ANO  INCLUDE 

APPROACH  TO  RESOLVE  REMAINING  UNCERTAINTIES  AND 
RISKS 

-  REALISTIC  FALL  BACK  ACTIONS  ANO  AL  T  F  RN  AT  I VF  S 

BUSINESS  PLANS  TO  SUPPORT  STRATEGY  CONTRACT 
TYPES  CONSISTS  NT  WITH  OBJECTIVES 

•  COMPE  TlTlVE  SELECTION  OF  SUBSYST  EMS 
CONSIDERATION  OF  F  X  1ST  INC.  MIL  I  TAR  Y  OR  COMMERCIAL 
HARDWARF  SOFTWARE 

JONS'DE  RATION  OF  FOREIGN  DEVELOPMENTS 

REQUIREMENTS  FOR  LONG  LEAO  PROCUREMENT  ANO 
INITIAL  LIMITED  PRODUCTION  COORDINATION  WHEN 
APPROPRIATE!  WITH  SINGLE  MANAGER  FOR 
CONVENTIONAL  AMMUNITION 

•  FIRM  ANO  REALISTIC  COST  pe  RFUHMANCE  ANO  SCHEDULE 
ESTIMATES,  AND  THRESHOLDS  ESTABLISHED 


,  •  Of  .  f  LOPMENT  PROGRESS  ANO  OT&fc  RESUL  TS  SUPPORT 

DECISION  TO  PROCEED 

l  COST  SCHEDULE  PERFORMANCE  ANO  SUPPOR  T  ABIL  1  T  * 

TRADE  OFFS  ACCEPTABiE 

;  DESIGN  TO  COST  ANO  Uf  E  CYCl  E  COST  REQUIREMENTS 

realistic 

j  SYSTEM  COST  EF  FECTlVE  AND  AFFORDABLE  REMAINS 

BEST  ALTERNATIVE 

!  MAJOR  PROBLEMS  lOENTlHEO. RESOLVED 

1  NATO  STANDARDIZATION  INTE  ROPE  RA8IL  ITY 

REQUIREMENTS  SATISFIED 

•  SUPPORT  SUBSYSTEMS  FOR  INITIAL  OPERATIONAL  uM  TS 
ASSESSED 

•  MINIMUM  OPE  RATIONAL  R&WMET  GOALS  ANO 
THRESHOLDS  ESTABLISHED  FOR  INITIAL  DEPLOYMENT 

1  •  MANPOWER  AND  TRAINING  REDuiPcMENTS  DEvELOPE  D 

I 

l 


•  ACQUISIT  ION  STRATEGY  ,;PU4»f  Q  AND  BE  ■  NO  E  *  S  u  T  t  Q 

1  •  PROGRAM  MANAGLMf  N’T  j  T  U  i_'C  T  R  l  AND  *>-  AN  SOUND 

I  AND  AOFOUATFL  v  SUPPORTED 

•  PRODUCTION  QUANTITY  A  E  Gu"  RE  ME  NTS  v  A  l.  D 

•PRODUCTION  PRODUC  1 8  >  L  TS  QU  A  L  .  T  y  ASSUH  ANC  f  AND 
FACILITIES  ISSUES  IDENTIFIED  MANAGE  D 

J  PRODUCTION  READINESS  REVIEW  COMF1  i  Tt 

I  -  REQUISITES  FOR  FUTURE  01  C  1  S'ONS  DEFINED 

I 

|  GOMPf  TIT  ION  CONSiQE  RE  D  l  ..  A  ■:  .  NO  SOU  “C  k 

j  •  PLANNING  FOR  DEPLQYMEN’  ADEQUATE 
MANPOWER  AND  TRAINING 
LOGISTICS  READINESS 

I  INTEGRATION  WITH  EXIST  iNG  S*STE  MS 


•  READINESS  TO  SUBMIT  INITIAL  SAFE 
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hoc  neon  ‘.1 sed  bv  th^  DSARC  as  the  orim.srv  i  3CTTi^,'.t  it :  v*1 
arriving  at  milestone  recommendations.  Converse  1%  ,  t h ** 
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w  a  s 


in  .1  contract- 1  i<e  capacity  similar  t  o  t::-  C  ’  ?/'?. 
reticaily,  the  TEMP  requires  or  ior  .nrrt«w»nr.  nn  tr.e  -.it  : 
mCi  -’cooe  or  the  t-euts  to  h«  cone ■;ct- ■:  ; nd  't  a;  ’ 

all  parties  the  T&E  iata  requirements  tor  eacr:  ,rjcc»®,i  i  n 
milestone  review.  The  T5.E  assessment  is  an  i  rv:*-por.'i»--nh  * 
after-the-fact  input  to  the  DSARC  ami  the  Secretary 
Defense  regarding  trie  TSE  accomplished  and  p  1  ?nn«s  i.  The 
SDDM,  among  other  things,  provides  guidance  and  direct!  a 
for  t.oe  next  phase  of  acquisition. 

Interviews  were  held  with  over  35  individuals  or.  r f. <•- 
OSD  staff,  at  the  Service  Headquarters,  in  pro'e-" 
offices,  and  in  test  and  evaluation  organization, 
some  cases,  a  formal  interview,  u.si.oq  a  struct  jr-'  i  .  u  ■  ■  ■ 
•w  guide,  was  conducted;  in  otr.er  cases,  informal  :  re¬ 
cessions  were  held.  Appendix  3  contains  a  copy  of  t r. o 
interview  guide  that  was  used  for  formal  interviews. 

4 .  ANALYSIS 

In  order  to  place  each  milestone  decision  in  its 
proper  Historical  perspective,  trends  in  acquisition  and 
TSE  policy  were  documented  and  a  diachronic  chart  sov.ri 
the  decade  of  the  1970s  was  oreDared.  This  chart,  whic- 
is  shown  in  Appendix  D  and  described  in  detail  in 
Chapter  III,  identifies  external  events  that  could  have 
arrecteo  individual  decisions.  The  process  of  relation 
such  events  to  their  historical  context  and  to  each  othe 
has  proven  to  be  a  valuable  research  technique  on  s imi 1  a 
stud ie  s . 

The  DSARC  decisions  were  treated  first  as  a  a  roue  in 
•which  trends  were  identified  and  relationships  with  the 
decision  environment  were  analyzed.  Then,  selected  Dc-\? 
decisions  were  analyzed  in  depth  to  discover  alternative 
reasons  for  conditional  decisions  and  tr.e  under  lyi  no 
causes  of  inadequate  TSE  data. 

★  ★  ★  ★  w 

The  fundamental  approach  used  in  this  study  was  to 
analyze  those  decisions  in  wnich  the  DSARC  nr  :  sc  ina! i  - 
die  a  ted  there  were  inadequate  TSE  data  in  the  context  •*  f 
tr.e  existing  decision  environment.  The  next  c:  act-' r  - 
scribes  chat  environment  in  terms  of  acquisition  col 
Ts£  do lie/  and  external  events. 


I C3TT 


ITT.  THK  PEC  I  S  T  'N  ENVIRONMENT 


The  i.'ciiion  environment  affects  milestone-  reviews  :  r. 
subt  I*1  rut  important  ways.  Policy  emphasis  swinas  r.etwe'- 
~>xt  rentes  of  caution  and  expediency  and  can  sometime-  fisc 
a  program  out  of  phase  with  the  dominant  pr.  1  losophy  at  tr. 
time  of  the  review.  High-level  personnel  change  as  part 
of  the  political  process  and  new  ideas  and  approaches  :,e- 
:ome  vogue  and  then  are  replaced  by  others.  DSAPC  deci¬ 
sionmakers  are  influenced  not  only  by  external  pressures 
from  Conaress,  the  GAO,  and  the  public  media,  but  also  oy 
specific  events  such  as  the  relianility  failures  of  t.ne 
J.S.  helicopters  in.  Iran  or  the  cancellation  of  the  B-i 
bomber.  Though  the  environment  is  complex  and  hiahiy  cou 
pled,  it  nevertheless  provides  clues  to  understanding  tr.e 
DSARC  decisions. 

This  chapter  traces  the  major  themes  in  acquisition 
and  TS.E  policy  since  about  1970.  In  addition,  a  dia¬ 
chronic  chart  is  used  to  provide  a  historical  perspective 
of  significant  events  and  trends. 

1  .  TRENDS  IN  ACQUISITION  POLICY 

The  T&E  revolution  which  occurred  in  the  early  19  7 Cs 
was  a  by-product  of  the  acquisition  policy  initiatives  of 
Deputy  Secretary  of  Defense  Packard.  Appendix  7  contains 
a  brief  overview  of  some  of  the  key  studies  ana  other 
events  since  1970  that  influenced  acquisition  policy. 
Several  important  trends  can  be  identified  which  affect 
the  questions  raised  in  this  study  regarding  adequacy  of 
T&E  data.  Specifically,  the~e  include: 

.  Centralization  of  operations  within  OSD 

reached  a  maximum  in  the  mid-  1960s,  but  r.as 
been  gradually  decreasing  since  the  early 
1970s  and  further  decentralization  can  be 
expected . 

.  The  acquisition  reforms  embodied  in  OMB  Cir 

cular  A - 1 0 9  began  in  the  early  19~0s,  may 
have  peaked  in  the  late  1970s,  and  now  arj 
leveling  off. 

.  The  role  of  T&E  in  the  decisionmaking  pro¬ 
cess  grew  rapidly  in  the  early  1970s  as  a 
companion  to  the  new  way  of  doina  business 
introduced  by  Packard.  However,  as  indi¬ 
cated  by  the  Defense  Science  Board,  the 


potential  exists  for  testing  to  increase  the 
lengtn  of  the  acquisition  cycle,  anti  the 
influence  of  T&E  in  the  decision  process  may 
also  level  off  as  the  desire  to  expedite 
development  increases. 

.  The  emphasis  on  operational  suitability  has 
risen  steadily  throughout  the  last  decade. 
Fueled  by  increases  in  weapon  system  com¬ 
plexity,  the  all-volunteer  force,  decreasing 
skills  of  enlisted  personnel,  and  revela¬ 
tions  about  U.S.  military  readiness,  it  will 
probablv  continue  to  do  so  throughout  the 
1980s.' 

.  Finally,  the  total  rejection  of  concurrent 
development  and  production  existing  since 
1970  has  given  wav  to  a  more  permissive  at¬ 
titude  in  which  decreases  in  acquisition 
time  are  traded  for  increases  m  risx  on  a 
case-bv-case  basis. 

Exhibit  III-l  shows  these  trends  on  a  subjective  scale 
increasing  on  the  y-axis.  They  provide  backaround  for  a 
discussion  of  the  evolution  of  T&E  policy. 

2 .  EVOLUTION  OF  T&E  POLICY 

Prior  to  the  1960s,  well-accepted  and  understood  in¬ 
stitutional  mechanisms  of  long  standing  within  the  Ser¬ 
vices  allowed  the  agencies  concerned  in  the  acquisition 
process  to  work  out  the  engineering  development  and  asso¬ 
ciated  testing  activities  oeyond  the  purview  of  higher 
authority.  When  a  product  was  ready,  the  appropriate  Ser¬ 
vice  testing  agency  formally  evaluated  the  system  for  op¬ 
erational  use.  Deficiencies  (if  any)  were  then  corrected 
ar.d  the  item  was  approved  for  Service  use  by  tne  Service 
Chief  or  Service  Secretary.  This  was  followed  by  quantity 
production. 

As  descrioed  in  the  previous  section  and  in 
Appendix  C,  during  the  decade  of  the  1960s  sweeping 
changes  to  the  acquisition  process  were  accompanied  by 
extraordinary  centralization  of  authority  and  decision¬ 
making.  The  growth  of  the  OSD  and  Service  headquarters 
staffs  in  size  and  involvement  in  materiel  matters  was 
accompanied  by  an  erosion  of  the  cadre  of  experienced 
technical  personnel  and  authority  of  the  Services'  own 
materiel  development  establishments.  In  former  years, 
these  establishments  managed  most  of  the  test  and  evalu¬ 
ation  materiel  acquisition  activities  as  they  saw  fit, 
often  beyond  the  effective  control  of  the  Service 
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Chiefs  or  even  the  Secretaries.  During  the  early  1970s 
and  continuing  to  the  present,  the  trend  has  been 
reversed.  The  policy  now  is  to  ensure  greater  decentral¬ 
ization  of  material  acquisition  authority  with  the  Secre¬ 
tary  of  Defense  retaining  only  four  key  milestone  de¬ 
cisions  for  major  programs. 

Exhibit  1 1 1 -  2  shows  the  changes  in  T&E  policy  that 
accompanied  the  larger  movement  in  acquisition  policy 
overall.  The  time  scale  is  divided  according  to  the  in¬ 
troduction  and  several  revisions  of  DOD  Directive  5000.3, 
i.e.,  pre-1970,  1970  to  mid-1975,  mid-1975  to  mid-1978, 
mid-1978  to  1979,  and  1979  to  the  present.  The  date  of 
the  operant  DOD  directive  for  each  period  generally  di¬ 
vides  the  time  span  in  recognition  of  the  fact  that  man¬ 
agement  philosophies  tend  to  emerge  and  even  be  put  into 
practice  somewhat  before  they  are  formally  introduced  as 
policy. 

(1)  Baseline 


Prior  to  1970,  the  role  of  T&E  in  the  acquisition 
process  was  stated  as  part  of  DOD  Directive  3200.9, 
Initiation  of  Engineering  and  Operational  System  De¬ 
velopment,  i.e.,  there  was  no  separate  DOD  directive 
on  T&E,  per  se.  Emphasis  at  the  time  was  on  total 
Package  Procurement,  and  there  was  no  need  to  focus  on 
T&E  from  a  decisionmaking  standpoint  because  formal 
periodic  reviews  of  major  systems  and  the  use  of  dem¬ 
onstrated  performance  as  the  pacing  function  had  not 
yet  been  introduced.  Responsibility  for  T&E  was  gen¬ 
erally  assigned  under  the  DDR&E  and  received  a  pre¬ 
dominantly  developmental  emphasis.  Although  test 
planning  was  included  as  part  of  the  Technical  Devel¬ 
opment  Plan  ( TDP)  prepared  by  the  Service,  the  OSD 
staff  frequently  were  involved  in  the  details  of  pro¬ 
gram  management. 

Development  T&E  was  judged  by  the  Blue  Ribbon 
Defense  Panel  as  being  generally  well  understood  and 
executed  as  part  of  the  system  design  process.  Opera¬ 
tional  T&E,  on  the  other  hand,  received  little  empha¬ 
sis  before  production;  it  was  judged  to  be  infrequent, 
poorly  designed  and  executed,  and  generally  inade¬ 
quate.  In  addition,  the  importance  of  realistic  test 
environments  that  closely  approximated  the  actual  op¬ 
erating  conditions  of  the  system  was  not  fully 
app  rec  l  a  ted  . 
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EVOLUTION  OF  TEST  & 


TIME  FRAME 

MAJOR  000 

BASIC  PRINCIPLES 

I - 

OSD  RESPONSIBILITIES 

TEST  PLANNING 

DIRECTIVE 

L _ _ 

Overall  acquisition 
policy  stated  m 

OOOO  3200  9 
No  local  000 
directives  on  Id 


£mpfias>s  on  *ntal  paihaqi-  pioiuiemrni  with  little  use  o! 
demonstiatnl  pvrturinannr  as  the  pacing  function  no 
lonnal  periodic  reviews 


T 4 1  responsibilities  generally  assigned  under  the 
0DR1E  received  a  predominantly  developmental 
focus 


Test  planning  included  as  a  chapter  m  the  Technical 
Development  Plan  TOP  OSO  frequently  involved  in 
detailed  direction  ot  programs 


0000  6000  3 
January  9  19/3* 
8  pages 


0000  6000  3 

M.d  1975  M.d  1  970  May  20  1  9  76r 

8  pages' 


Disciplined  use  nl  I  it  I  results  III  assist  •»>  redu 

•  Condut  t  T  &E  early 

•  Base  schedules  .in  act  iiinpl  shiny  FAE  pimi 
mileslune  dec  sums 

•  Oetine  need  mil  estimate  military  utility  pnu 

Vitiating  itevi  lopment 


OOTAE  established  and  fli  s-gnated  as  havu.g 
across  the  board  lespuns-biUty  lor  OSD  m  l&E  in 
Provides  to  SECDEf  and  DSARC  independent 
assessment  uf  T&t  accomplished  amt  planned 


I  he  000  component  prepares  an  nvrrall  integrated  FAE 
plan  as  early  as  possible  no  OSO  approval  required'  The 
plan  is  to  be  kept  current 


;  The  test  plan  is  now  identified  as  a  separate  document 
1  called  a  Test  Master  Plan  TEMPI  Required  approval  is 
I  ambiguous  Changes  are  to  be  documented 


Role  o'  laE  elaborated  and  eipanded 

•  Increased  emphasis  on  operational  effectiveness  jnd 
000D  bOOO  3  I  suitability 

Mid  19  78  19/9  April  1  i  19/8  •  Issues  test  cnten.i  and  measures  of  effectiveness 

1 16  page-.  established  before  tests  bpqm 

•  Allow  fur  r  unlmgeni  y  testing 
I  •  Minim  ire  subjective  judgment 


Staff  responsibilities  within  OSO  split 

•  DDTAlfnrOTAE  and  coordinates  OT  AE  aspects 

•  USD  i  PAA  E '  (oi  OT 4(  matteis 


T  E  MP  to  be  submitted  tn  OOTAE  for  coordination  review 
relative  to  major  milestones  but  an  approved  TEMP  is 
a  prerequisite  for  Milestone  II  Changes  are  to  be 
properly  documented 


j  Essentially  the  same 

•  Early  OTAE  and  OTAE  to  reduce  risks  and  estimate 

DODO  6000  3  !  operational  potential 

December  26  19/9  I  •  Issues  objectives  and  criteria  before  test  begins 
1 1  8  pages)  !  relate  lo  mission  need 

1  •  Successful  TiE  required  before  committing  resources 
or  progressing  to  next  phase 

•  Minimite  subjective  judgment 


OOTAE  (again)  given  total  responsibility  for  IAE 

•  Policy  and  interservice  problems 

•  TAE  assessments 

•  Joint  testing 

•  Major  range  and  test  lacilitirs  base 


TEMP  must  be  submitted  for  OSO  approval  prior  to 
Milestone  f  -  guidelines  for  preparation  included  in 
6000  3  Changes  to  be  sported  promptly 


While  the  policies  shown  were  formally  introduced  *nh  the  first  publication  of  0000  6000  3  much  of  then  substance  was  issued  earlier  by  DEPSECOEE  multi  addressed  memoranda 
Conduct  of  Operational  Test  end  Evaluation  February  11  19/1 
Role  of  00R8E  in  Test  and  evaluation  as  Related  to  OCP  System  April  21  1971 
Test  and  Evaluation  m  the  System  Acquisition  Process  August  3.  1971 
These  changes  were  formally  made  as  6000  3  Chi.  April  19/4 
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TEST  &  EVALUATION  POLICY 


DEVELOPMENTAL  TIE 


OPERATIONAL  T&E 


INDEPENDENT  EVALUATION 


CUMBiNEO  DT  ANO  fl! 


r  in  tha  Technical 
pMntly  involved  in 


Assessed  by  the  Blue  Ribbon  Defense  Panel  as 
generally  well  understood  and  executed 


W«rall  integrated  T&E 
pi  oval  required)  The 


little  emphasis  on  operational  testing  betora  pioduction 
OT  was  infrequent  poorly  designed  and  executed  and 
generally  inadequate 

Importance  oE  realistic  test  environments  approximating 
actual  operating  conditions  not  generally  recognized 


No  specific  policy  regarding  independent  evaluati  on 
developing  and  01  using  agencies  assess  peiformanrr  ol 
new  weapon  systems 


Conducted  to  demonstrate  design  process  is  complete 
risks  have  been  minimized  system  will  meet 
Specifications 

•  Planned  conducted  monitored  by  developing  agency 

•  Start  early  include  components  subsystems 
prototypes  and  preproduction  models 

•  Specific  obiectives  for  each  phase 


Importance  of  operational  testing  begins  to  emerge 
Initial  OT  A  E  ilOT&E)  to  be  conducted  before  maior 
production  decision 

lOT&E  to  be  m  as  realistic  an  environment  as  possible  and 
conducted  by  personnel  like  those  expected  to  use  and 
maintain  the  system 

Generally  pilot  production  items  to  be  amployed 


Each  service  is  to  piovide  one  101  a  limited  numbe  maid 
field  agency  responsible  for  GT&E  separate  and  distinct 
from  developing  procuring  command  Also  an  OT&E  local 
point  to  be  established  with.n  headquarters  staff 


Mfirate  document  • 

W  Required  approval  is  •  SAME  AS  ABOVE 

IM«t«d 


;  Emphasis  on  earlier  OT&E  also  preproduction 
j  prototypes  rather  than  pilot  production  items  are 
I  generally  to  be  used  tor  I0T&E 


Policy  changed  such  that  unly  onr  major  field  agency 
responsible  for  0T&E  is  permitted  must  be  separate  fium 
using  command  as  well  as  developing  procuring 


A  frequeni  pract.rr  largr  .  due  in  urgency  to  deploy 
and  resource  limitations  fur  (JHE  Consistent  with 
0S0  emp.iasis  on  concurrency  ol  development  and 
production 


OT  should  be  separate  from  OT  however  may  be 
combined  with  early  OT  when  appio.  i.ate 


SAME  AS  ABOVE 


+ 
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»  coordination  review  1  Essentially  the  same  adds  Phase  0  and  post  Milestone  lll| 
I  "approved  TEMP  is  (  testing,  stresses  importance  ol  software  and  provides  I 
kMfts  are  to  be  j  checklist  ol  ’'titties' 


Essentially  the  same  specifically  adds  enemy 
countermeasures  as  pari  of  a  realistic  environment  Also 
Stales  that  preproduction  items  or  full  scale  engineering 
models  to  be  used  tor  lOT&E  if  (hey  are  reasonably 
representative  Defines  OTiE  obiectives  for  each  phase 


Position  softened  somewhat  geneia’ly  to  be  conducted 
separately  but  may  br  combined  when  .  (early  identified 

Essentially  the  same  mnre  thorough  definition  of  thr  cusl  schedule  bene*. ts  or  nsb  acceptable  However 

role  and  responsibilities  of  the  independent  OT  ft  E  agency  norma!  practice  is  that  01  supporting  a  production 

I  decision  writ  be  conducted  independently 


^pcoval  prior  to 
MNlpn  included  in 
IMMTlY 


I  Essentially  the  same  adds  that  the  development  agency 
I  wilt  support  OT&E  obiectives  as  appropriate 


Essentially  the  same  adds  the  obtcclive  to  obtain  a  valid 
estimate  of  (he  user  s  capability  to  operate  and  maintain 
the  system  in  peacetime  and  wartime 


ESSENTIAllY  THE  SAME 


Specifically  highlighted  that  DT  and  OT  may  be  combined 
but  that  the  OT&E  agency  will  be  a  lull  pariner  and  that  OT 
credibility  will  be  maintained  final  testing  prior  to 
Milestone  III  will  emphasize  operational  testing 


Mhi  addressed  memoranda 


EXHIBIT  III  2 

EVOLUTION  OF  T8.E  POLICY 
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Development  testing  (DT)  and  operational  testing 
(OT)  were  frequently  combined  due  to  a  perceived  ur¬ 
gency  to  deploy  new  systems  and  to  resource  limita¬ 
tions  for  operational  test  and  evaluation  (OT&E) . 

This  was  consistent  with  the  existing  philosophy  of 
concurrent  development  and  production.  There  were  no 
specific  policies  regarding  independent  evaluation  of 
test  results,  and  assessments  of  new  systems  were 
usually  performed  by  the  developing  or  using  agencies. 

By  today's  standards,  Congress  was  hardly  in¬ 
volved  in  the  acquisition  process  although  they  were 
becoming  increasingly  concerned  with  chronic  cost 
overruns  and  performance  deficiencies. 

In  1971,  Secretary  Packard  issued  three  memoranda 
to  the  Service  Secretaries  on  T&E  which  were  consis¬ 
tent  with  his  earlier  initiatives  on  overall  acquisi¬ 
tion  policy.  These  led  to  the  first  publication  of 
DOD  Directive  5000.3  in  January  1973.  Since  then  the 
directive  has  been  revised  three  times  in  response  to 
changing  emphasis  in  acquisition  policy  and  in  the 
perceptions  of  the  role  of  T&E. 

( 2 )  Basic  Principles 

The  first  major  change  in  basic  principles  called 
for  the  disciplined  use  of  T&E  results  to  assist  in 
reducing  risks  associated  with  moving  from  one  phase 
of  the  acquisition  cycle  to  the  next.  T&E  was  to  be 
conducted  early  and  development  schedules  were  to  be 
based  on  accomplishing  the  required  T&E  prior  to  key 
milestone  decisions.  These  basic  principles  continue 
today  except  that  the  role  of  T&E  has  been  further 
defined  and  expanded.  Emphasis  is  placed  on  early 
development,  test,  and  evaluation  ( DT&E )  and  OT&E, 
particularly  to  evaluate  operational  effectiveness  and 
suitability.  Issues,  objectives,  and  criteria  relat¬ 
ing  to  mission  needs  are  to  be  defined  in  advance;  anc 
subjective  judgment  is  to  be  minimized. 

In  1973,  the  Deputy  Director  (Test  and  Evalua¬ 
tion)  was  created  within  the  DDR&E  to  assume  across- 
the-board  responsibility  for  T&E  matters.  This  has 
continued  until  the  present  except  for  the  period 
April  1978  through  December  1979  when  staff  responsi¬ 
bilities  within  OSD  were  split:  the  DDT&E  had  respon¬ 
sibility  for  DT&E  and  for  overall  coordination  of  T&E, 
but  the  ASD,  Program  Analysis  and  Evaluation  (PA&E) 
had  direct  responsibility  for  OT&E.  This  policy  was 
reversed  in  January  1980.  Today,  the  DDT&E  re¬ 
sponsibilities  include: 


.  Formulating  DOD  T&E  policy  and  resolving 

inter-Service  problems 

.  Conducting  independent  T&E  assessments  of 

Service  programs  to  be  provided  directly  to 
the  Secretary  of  Defense  as  well  as  to  the 
DS  ARC 

.  Sponsoring  joint  tests  to  evaluate  opera¬ 
tional  effectivess  of  systems  which  cut 
across  Service  lines 

Managing  the  Major  Range  and  Test  Facilities 
Base  (MRTFB) . 

'  3 )  Test  Planning 


In  the  original  version  of  DODD  -000.3,  each  DOD 
component  was  to  prepare  an  overall  integrated  T&E 
plan  as  early  as  possible  but  there  was  no  requirement 
for  the  Services  to  obtain  OSD  review  or  approval.  In 
1975,  the  plan  was  identified  as  a  separate  formal 
document,  called  a  TEMP,  but  the  requirements  for  re¬ 
view  and  approval  were  still  ambiguous,  partly  because 
of  Service  objections  to  what  they  believed  was  addi¬ 
tional  OSD  control  of  their  programs.  The  ambiguity 
continued  through  1978  when  the  Services  were  required 
to  submit  their  TEMPs  to  DDT&E  only  for  coordination/ 
review,  while  at  the  same  time  approval  somehow  was 
required  prior  to  milestone  II.  Full  clarification 
occurred  in  the  latest  revision  of  5000.3:  TEMPs  must 
now  be  submitted  to  OSD  for  approal  prior  to 
milestone  I,  and  detailed  guidelines  for  TEMP  prepara¬ 
tion  are  included  in  the  directive.  Changes  to  T&E 
plans  are  to  be  reported  promptly  to  DDT&E. 

( 4 )  Development  T&E 

DT&E  has  changed  little  over  the  last  decade. 

The  objective  of  DT&E  continues  to  be  to  demonstrate 
that  the  design  process  is  complete,  that  risks  have 
been  minimized,  and  that  the  system  will  meet  its 
design  specifications.  DT&E  is  normally  planned,  con¬ 
ducted,  and  monitored  by  the  developing  agency  and  has 
specific  objectives  for  each  acquisition  phase. 

Earlier  DT&E  has  been  stressed  in  accordance  with  the 
introduction  of  Phase  0  as  has  post-milestone  III 
testing  with  regard  to  evaluating  proposed  system 
modifications.  The  importance  of  evaluating  software 
and  reliability,  a va i labl i 1 i ty ,  and  maintainability 
(RAM)  during  DT  has  recently  been  emphasized,  and  the 
developing  agencies  have  been  asked  to  support  OT&E 
objectives  to  the  maximum  extent. 
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(  5 )  Operational  Tt»E 

The  original  DODD  5000.3  stated  that  it  was  im¬ 
portant  to  complete  initial  operational  test  and 
evaluation  ( IOT&E)  before  the  major  production  deci¬ 
sion;  that  IOT&E  was  to  be  conducted  in  as  realistic 
an  environment  as  possible;  and  that  the  system  was  to 
be  operated  and  maintained  by  typical  Service  oer- 
sonnel.  Subsequent  revisions  required  earlier  OT&E 
and  were  more  specific  with  regard  to  the  objectives 
of  OT& E  during  each  acquisition  phase.  Tne  type  of 
hardware  to  be  employed  for  IOT&E  has  gradually 
changed  from  pilot  production,  to  preprocuct ion  proto¬ 
type,  to  full-scale  engineering  models  as  long  as  they 
are  reasonably  representative  of  the  production  con¬ 
figuration.  This  trend  is  somewhat  inconsistent  with 
the  acquisition  policy  that  developed  during  the  same 
period  which  allowed  limited  production  for  OT&E  after 
milestone  II. 

( 6)  Independent  Evaluation 


Independent  evaluation  was  one  of  the  key  recom¬ 
mendations  of  the  Blue  Ribbon  Defense  Panel  and  was 
perhaps  the  major  innovation  of  the  T&E  policies  in¬ 
troduced  with  5000.3.  Originally,  each  service  was 
required  to  establish  one  (or  a  limited  numoer  of) 
field  agencies  responsible  for  OT&E.  These  agencies 
were  separate  and  distinct  from  the  developing/ 
procuring  command  and  also  were  able  to  establish 
strong  headquarters  staff  focal  points  to  assist  in 
conducting  required  test  and  evaluation.  in  1975,  the 
policy  was  changed  so  that  only  one  field  agency  was 
permitted.  in  addition,  it  had  to  be  separate  from 
the  using  command  as  well  as  the  developing /procu r i ng 
command.  The  theory  was  that  both  users  and  develop¬ 
ers  had  their  own  unique  biases  which  would  tend  to 
affect  their  evaluation. 

(7)  Combined  DT  and  OT 


The  practice  of  combining  DT  and  OT  was  offi¬ 
cially  proscribed  during  the  early  1970s.  The  manage¬ 
ment  philosophy  then  in  effect  was  that  effective  OT 
required  exercising  the  system  over  the  full  range  or 
expected  operating  conditions,  in  conjunction  with 
other  systems  with  which  it  must  function,  and  under 
the  most  realistic  conditions  possible.  It  was  oe- 
lieved  that  limited  and  carefully  controlled  environ¬ 
mental  conditions,  operation  and  maintenance  by 
techn ic ian - leve 1  personnel,  and  added  instrumentation 
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commonly  imposed  by  the  development  tester  prevented  a 
true  operational  test  and  contaminated  the  test  re¬ 
sults.  While  separate  DT  and  OT  are  still  thought  to 
be  most  desirable,  time  and  cost  pressures  have  gradu¬ 
ally  altered  the  official  policy.  In  1978,  DT  and  OT 
were  generally  to  be  conducted  separately  but  could  be 
combined  when  clearly  identified  cost/schedule  bene¬ 
fits  or  risk  is  acceptable.  The  normal  practice,  how¬ 
ever,  was  for  the  OT  to  be  conducted  independently  in 
support  of  a  production  decision.  The  1979  revision 
to  5000.3  states  that  DT  and  OT  may  be  combined  when 
appropriate  but  that,  in  such  cases,  the  OT&E  agency 
will  be  a  full  partner  and  that  OT  credibility  will  be 
maintained.  Final  testing  prior  to  milestone  III  will 
now  emphasize  operational  testinq. 

( 8 )  Summary 

Exhibit  III-3  shows  selected  trends  in  T&E  policy 
in  a  fashion  similar  to  that  in  which  Exhibit  1 1 1 -  2 
showed  trends  in  overall  acquisition  policy.  Trends 
in  T&E  policy  may  be  summarized  as  follows: 

.  The  most  significant  trend  is  the  growing 
influence  of  the  DDT&E  in  DSARC  decision¬ 
making.  Part  of  this  influence  derives  from 
clearer  authority  granted  through  changes  in 
high-level  acquisition  directives  and  in¬ 
struction  such  as  5000.1,  5000.2,  and 
5000.3.  There  also  is  an  indication  that 
the  DSARC  principals  may  have  come  to  re¬ 
spect  DDT&E  independent  assessment  of  test 
results  and  test  planning  and  have,  in  fact, 
become  dependent  for  DDT&E  identification  of 
performance  and  operational  suitability  is¬ 
sues. 

Similarly,  the  Services  in  recent  years  have 
recognized  the  importance  of  early  and 
continuous  communication  with  the  DDT&E,  not 
only  because  of  possible  effects  on  their 
programs,  but  also  because  of  their  belief 
that  a  better  test  program  and  a  better 
product  will  result.  The  increased  slope  of 
the  DDT&E  influence  curve  after  about  1978 
reflects  the  vigorous  activites  of  the 
office  to  institute  stricter  control  and  use 
of  the  TEMP,  to  emphasize  the  importance 
of  assessing  operational  suitability  early 
in  the  life  cycle,  and  to  increase  the  use 
on  analytical  techniques.  The  office  is 
presently  regarded  as  both  competent  and 
fair  by  OSD  and  the  Services  alike. 
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PROSCRIPTION  ON  COMBINED  DT  &  OT 


The  importance  of  OTM\  nau  been  stf-a«  i  i  v 
increasing  tnrougnout  tr<e  -iecaue .  Tri i s 
consistent  with  tne  empnasis  on  operation..:, 
suitability  noted  earlier.  Tne  present  view 
is  that,  short  or  actual  warfare,  opera¬ 
tional  testing  is  the  only  legitimate  means 
of  assessing  how  a  system  will  perform  in 
its  operational  environment.  The  trend  ac¬ 
knowledges  the  difficulty  in  establishing 
technical  system  specifications  and  the  fact 
that  meeting  them  (generally  as  evaluated  in 
DT)  does  not  necessarily  mean  tne  system 
will  perform  its  intended  function.  It 
should  be  noted,  however,  that  policy  r,uS 
bacKed  away  from  requiring  OTf>E  on  produc¬ 
tion  or  preproduction  hardware  prior  to  tne 
milestone  III  review  ana  that  combining  n't 
and  OT  is  now  accepted  under  some  circum¬ 
stances  as  a  means  of  accelerating  tne 
acquisition  process  (see  below). 

Running  somewhat  counter  to  the  emphasis  on 
OTi E  is  an  apparent  softening  of  tne  atti¬ 
tude  on  the  deleterious  effects  of  combining 
DT  and  OT.  However,  this  trend  does  paral¬ 
lel  the  emerging  philosophy  on  concurrency, 
which,  in  fact,  reflects  a  pervaswe  trend 
throughout  acquisition  policy:  that  abso¬ 
lute  or  cook  boo  k  approacnes  are  not  opti¬ 
mally  effective;  rather,  that  the  process 
should  be  tailored  to  suit  the  individual 
circumstances.  In  this  case,  it  an  acknow¬ 
ledgement  that  Ti»E  activities  can  oe  reduced 
when  risks  permit. 

While  the  concept  of  formal  T&E  planning  in 
the  context  of  decentralized  operations  was 
introduced  in  the  1970-1971  time  frame,  tne 
TEMP  did  not  become  a  significant  planning 
tool  until  the  end  of  the  1970s.  Tne  ex¬ 
planation  lies  in  the  initial  reluctance  of 
the  Services  to  allow  OSD  such  a  detailed 
role  in  the  management  of  their  programs. 
This  also  explains  the  ambiguity  of  early 
policy  statements  on  the  purpose  of  the  doc¬ 
ument.  According  to  an  OSD  study,  until 
early  1979  the  TEMP  was  treated  casually 
even  by  DDT&E.  Subsequent  recognition  tnat 
it  needed  to  be  elevated  in  stature  and  con¬ 
tent  led  to  the  changes  in  5000.3  discussed 
earlier.  Now,  the  status  of  TEMPS  are 
tracked  much  more  carefully,  and  DDT&E  main¬ 
tains  statist ic s  on  cu  rrency ,  approval,  re¬ 
visions,  etc. 
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Although  not  a  trend,  an  important  event  in  the  evolu¬ 
tion  of  T&E  policy  occurred  in  1974  when  changes  were  made 
to  the  procedures  by  which  the  major  ranges  and  test 
facilities  were  reimbursed  for  services  provided  to  indi¬ 
vidual  users.  Previously,  the  ranges  had  been  predomi¬ 
nantly  institutionally  funded  from  their  own  Operations 
and  Maintenance  (O&M)  lines  of  the  DOD  budget.  Most  serv¬ 
ices  were  provided  at  no  cost  to  the  specific  programs. 
Now,  in  accordance  with  the  Uniform  Funding  Policy,  users 
are  required  to  reimburse  the  test  activity  for  all  direct 
costs  that  can  be  associated  with  a  particular  program; 
only  general  support  costs  are  funded  by  O&M  accounts. 

While  the  Uniform  Funding  Policy  does  not  change  the 
overall  cost  of  testina  within  DOD,  it  does  shift  the  bur¬ 
den  from  those  preparing  and  •’ujti  tying  the  O&M  nudget  to 
those  advocating  specific  weapon  systems.  In  general,  the 
increased  cost  of  doing  business  from  the  point  of  view  of 
the  program  manager  has  not  been  reflected  m  increases  to 
program  budgets.  This  has  added  to  the  difficulty  in  con¬ 
ducting  thorough,  comprehensive  test  programs  and  has 
encouraged  PMs  to  make  trade-offs  between  the  cost  of 
unplanned  development  problems  and  the  cost  of  planned 
testing. 

3 .  HISTORICAL  PERSPECTIVE 

Overviews  of  acquisition  reform  and  evolution  of  T&E 
policy  provide  one  perspective  for  evaluating  the  DSARC 
decisions  that  took  place  during  the  period  of  interest. 
Another  perspective  can  be  gained  by  putting  policy  events 
along  with  other  events  in  a  time-phased  display  called  a 
diachronic  chart.  Such  a  chart  for  the  period  1970 
through  1981  is  shown  in  Appendix  D.  It  shows  the 
f o 1  low  i  ng  : 

.  The  names  and  terms  of  senior  Administration  of¬ 
ficials 

.  Landmark  policy  events,  such  as  the  President's 

Blue  Ribbon  Defense  Panel  and  the  issuance  of  OMB 
Circular  A-109 

Acquisition-  and  T&E-related  directives  and 
i  ns  t  rue  t  ion  s 

Significant  elements  of  the  OSD  decisionmaking 
process  particularly  as  it  relates  to  T&E 

.  Selected  T&E  studies  conducted  by  Government 
agencies  and  outside  task  forces 
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.  GAO  reports  related  to  T&E  in  General  and  the 

effectiveness  of  DOD  system  acquisition  decision 
pract  ices 

Events  relating  to  Congressional  involvement  in 
T&E  aspects  of  the  acquisition  process 

.  Selected  defense-related  world  events 

.  The  names  and  terms  of  the  DSAPC  principals. 

The  diachronoic  chart  is  best  used  as  a  reference  on 
which  to  overlay  other  data.  However,  some  observations 
can  be  made  about  the  information  contained  in  the  c.oart 
itself  beyond  those  already  discussed  on  acquisition  and 
T&E  policy. 

The  U.S.  political  process  causes  a  turnover  in  key 
personnel  at  4-  or  8-vear  intervals,  notwithstanding  othe 
changes.  This  means  that  the  management  philosophy  or 
emphas i s  espoused  by  one  administration  may  not  continue 
into,  or  may  even  be  reversed  by,  the  next.  It  is  obviou 
that  establishing  a  stable  framework  for  acquisition  de¬ 
cisionmaking  under  these  circumstances  is  difficult.  Fur 
thermore,  the  turnover  in  DSARC  principals  from  one  mile¬ 
stone  to  the  next  on  a  given  program  tends  to  mai<e  eacn 
review  unique  rather  than  an  update  of  previous  decisions 
and  strategies.*  The  importance  of  the  DOD  career  civil 
servants  in  providing  continuity  under  these  circumstance 
should  not  be  underestimated. 

Another  significant  point  reflected  in  the  chart  is 
that  Congressional  and  GAO  interest  in  detailed  acquisi¬ 
tion  issues  and  decisions,  especially  as  they  relate  to 
T&E,  has  risen  steadily  over  the  decade.  In  1971,  Con¬ 
gress  passed  Public  Law  92-156  which  required  the  Secre¬ 
tary  of  Defense  to  submit  to  Congress  each  year,  beginnm 
in  FY73,  "...  data  on  operational  testing  and  evaluation 
for  each  weapon  system  for  which  funds  for  procurement  ar 
requested  (other  then  funds  requested  only  for  the  pro¬ 
curement  of  units  for  operational  testing  and  evaluation 


Data  from  the  final  report  on  Weapons  System  Costs 
from  the  DSARC  Cost  Reduction  Working  Group  (the 
"Little  Four"),  December  19,  1972,  indicate  that  the 
participation  by  the  same  DSARC  principal  on  11  repea 
or  successive  DSARCs  held  between  September  1969  and 
June  1971  ranged  from  64  to  less  than  10  percent. 

This  inconsistency  in  attendance  was  caused  by  substi 
tut  ion  for  the  DSARC  principal  rather  than  changes  in 
appo i ntment . 
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and/or  long  lead-time  items)."  These  data,  and  other  in¬ 
formation,  are  included  in  Congressional  data  sheets  which 
are  submitted  each  year  with  the  President's  budget. 

In  1975,  Congress  enacted  Public  Law  94-106  which  es¬ 
tablished  the  Selected  Acquisition  Report  (SAR)  as  a  legal 
reporting  requirement  of  the  DOD  for  designated  systems. 
Originally,  the  SARs  were  instituted  as  an  internal  DOD 
mechanism  for  keeping  the  ASD  Comptroller  apprised  of  the 
progress  of  selected  acquisitions  and  comparing  this  prog¬ 
ress  with  technical,  schedule,  and  cost  plans.  In  1969, 
in  response  to  a  request  from  the  Chairman  of  the  Senate 
Armed  Services  Committee,  the  Secretary  of  Defense  agreed 
to  provide  this  information  to  Congress.  Today,  the  SARs 
are  used  primarily  by  both  the  Senate  and  House  Armed  Ser¬ 
vices  and  Appropriation  Committees,  the  House  and  Senate 
Budget  Committees,  and  the  House  Government  Operations 
Committee.  The  SAR  is  also  sent  to  the  GAO,  OMB,  and 
other  agencies. 

According  to  unpublished  data  assembled  by  the  Direc¬ 
tor,  Management  Information  Control  and  Analysis,  ASD(C), 
Congress  indeed  is  playing  a  stronger  role  in  the  acquisi¬ 
tion  process  as  evidenced  by  almost  every  conceivable  mea¬ 
sure,  such  as  total  number  of  employees,  professional 
staff  members  on  defense- re lated  committees,  witnesses 
testifying  before  committees,  line  items  in  the  budget, 
and  number  of  pages  in  the  defense  authorization  and  ap¬ 
propriation  committee  reports.  In  addition,  the  GAO  since 
fiscal  year  1960  has  nearly  tripled  the  number  of  defense- 
related  reports  they  send  to  Congress  each  year.  As  seen 
in  the  diachronic  chart,  GAO  reports  on  T&E,  acquisition 
decisionmaking,  and  relations  with  Congress  have  increased 
steadily  since  1972.  Not  shown  on  the  chart  are  the 
weapon-specific  reports  prepared  by  the  GAO  at  the  begin¬ 
ning  of  each  calendar  year,  although  these  reports  are 
shown  on  "mini-d iachronics"  for  each  program  discussed  in 
Chapter  V. 

Another  important  aspect  of  the  diachronic  chart  is 
its  ability  to  show  the  connectivity  between  various 
events.  Sometimes  this  is  obvious,  as  when  a  revision  of 
a  directive  follows  its  original  publication.  Other  cases 
are  more  subtle,  such  as  the  effect  of  a  major  study  or 
report  of  a  commission  on  subsequent  revisions  to  policy. 
This  process  also  shows  that  sometimes  the  policy  change 
actually  occurred  before  the  landmark  event  which  pre¬ 
sumably  caused  it;  that  is,  the  landmark  event  was  actu¬ 
ally  confirming  what  had  generally  become  accepted  rather 
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than  breaking  new  ground  in  itself.  Where  connectivity 
between  events  can  be  inferred  and  is  practical  to  dis¬ 
play,  it  is  shown  as  a  dotted  line. 

★  ★  *  ★  ★ 

The  decision  environment  provides  a  backdrop  against 
which  to  analyze  the  38  DSARC  II  and  III  reviews  held  dur¬ 
ing  the  period  under  study.  The  results  of  these  DSARCs 
are  described  in  a  macroscopic  sense  in  the  next  chapter. 
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IV.  DSARC  DECISIONS 


There  have  been  nearly  250  DSARCs  since  the  process 
was  instituted  in  1969.  As  shown  in  Exhibit  IV-1,  there 
are  about  15  to  20  DSARCs  held  each  year,  with  a  peak  of 
30  in  1975.  This  chapter  addresses  the  38  DSARC  II  and 
III  reviews  held  between  January  1977  and  August  1980.  It 
includes : 


.  An  overview  of  the  milestone  II  and  ill  reviews 

The  identif ication  and  characterization  of  the 
decisions  made  by  the  DSARC 

.  An  analysis  of  trends  and  relationships  exhibited 

.  Identification  of  the  apparent  reasons  for  each 
dec  ision 

.  A  rationale  for  selection  of  programs  for  further 
study . 

1 .  OVERVIEW 

As  shown  in  Exhibit  IV-2,  there  were  38  milestone  II 
and  III  reviews  which  addressed  34  different  weapon  sys¬ 
tems  during  the  period  under  study.  The  dotted  lines  in¬ 
dicate  that  the  air-launched  cruise  missile  ( ALCM)  program 
had  both  milestone  II  and  III  reviews,  the  F-16  had  both 
III A  and  1 1 1 B  reviews,  and  the  high-speed  antiradiation 
missile  (HARM)  and  MX  programs  had  IIA  and  IIB  reviews. 
Follow-on  reviews  were  planned  for  the  major  caliber 
lightweight  gun  (MCLWG) ,  Roland,  and  XM-1  programs,  but 
they  did  not  occur  during  this  period. 

The  table  below  shows  the  distribution  of  reviews  by 
Service  and  milestone. 


'service'\^ 

MILESTONE  II 
REVIEWS 

MILESTONE  III 
REVIEWS 

TOTAL 

ARMY 

3 

9 

12 

NAVY 

5 

5 

10 

AIR  FORCE 

6 

8 

14 

JOINT 

2 

2 

TOTAL 

16 

22 

38 

Of  the  38  DSARCs,  12  addressed  Army  programs,  10  Navy 
programs,  14  Air  Force  programs,  and  2  joint  programs. 

The  ioint  programs  included  the  airborne  self  projection- 
jammer  (Navy/AF)  and  the  NAVSTAR  glooal  positioning  system 
(GPS)  program  (Na vy/AF./Army )  .  The  table  also  shows  that 
16  of  the  38  DSARCs  were  milestone  II  decisions  and  the 
remaining  22  were  milestone  III  decisions. 

2.  CHARACTERIZATION 


In  general,  DSARC  decisions  fall  into  three  basic 
categories:  unconditional  approval  to  proceed  to  the  next 

phase,  approval  to  proceed  to  the  next  phase  with  certain 
qualifications,  and  disapproval  (cancellation)  or  delay  in 
proceeding  to  the  next  phase  pending  resolution  of  techni¬ 
cal  or  other  issues  raised  by  the  DSARC.  In  the  case  of 
tae  production  decision  (milestone  III),  a  fourtn  cate¬ 
gory,  limited  production,  is  often  employed  and  is  helpful 
in  understanding  the  concerns  of  the-  DSARC  principals  and 
in  analyzing  trends  in  acquisition  policy. 

( 1 )  Milestone  II  Reviews 


The  16  milestone  II  reviews  are  shown  in  the 
table  below  by  Service  and  type  of  decision. 


''^DECISIONS 

UNCONDITIONAL 

APPROVAL 

DEFER/ 

SERVICE"-^ 

APPROVAL 

WlQUAllFICATIONS 

DELAY 

TOTAL 

ARMY 

2 

1 

0 

3 

NAVY 

2 

2 

1 

5 

AIR  FORCE 

2 

3 

1 

6 

JOINT 

0 

2 

0 

2 

TOTAL 

6 

8 

2 

16 

Six  DSARCs  (38  percent)  gave  unconditional  approval  to 
proceed  to  FSD:  ALCM ,  AV-8B,  laser  guided  projectile, 
DIVAD,  IUS,  and  Pershinq  II.  These  systems  had  com¬ 
pleted  their  test  programs  for  the  demonstration  and 
validation  phase  successfully  and  had  met  all  thres¬ 
holds  substantially.  No  significant  T&E  issues  had 
been  identified  and  no  additional  aadvanced  develop¬ 
ment  (AD)  testing  was  recommended.  Eight  (50  percent) 
of  the  programs  reviewed  at  milestone  II  received  ap¬ 
proval  to  proceed  to  FSD  with  certain  qualifications. 
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In  these  cases,  the  test  data  and  other  in  form a  ti  n 
provided  to  the  DSARO  were  regarded  us  adequate  to 
support  entry  into  FSD  if  additional  steps  wer»-  to  <•  r. 
to  reduce  risks;  i.e.,  the  risKS  involved  in  proceed¬ 
ing  were  considered  to  be  acceptable  in  the  context  of 
the  entire  body  of  existing  data  or  when  weigneo 
against  an  alternative  to  delay  or  cancel  the  pro¬ 
gram.  The  qualifications  imposed  were  usually  eiti.er 
to  correct  all  identified  decifiencies  and  demonstrate 
the  successful  correction  by  additional  testing  at  a 
later  stage  or  to  increase  confidence  in  certain  spe¬ 
cific  areas  by  providing  additional  relevant  test 
data.  In  two  cases  (12  percent) ,  HARM  and  MX,  it  was 
decided  to  delay  the  decision  to  proceed  to  FSD;  i.e., 
a  decision  was  made  to  remain  in  the  advanced  develop¬ 
ment  phase  until  either  specific  problems  were  re¬ 
solved  as  demonstrated  by  additional  testing  or  the 
program  was  redirected,  depending  on  the  case  uncer 
consideration. 

It  is  interesting  to  observe  that  while  the  Ser¬ 
vices'  programs  were  fairly  evenly  distributed  over 
the  decision  categories,  neither  of  the  two  joint  pro¬ 
grams  received  unconditional  approval. 

(2)  Milestone  III  Reviews 


The  22  Milestone  III  decisions  were  distributed 
by  Service  and  type  of  decision  as  shown  below. 


'''\0ECISI0N 

UNCONDITIONAL 

FULL  SCALE 
PRODUCTION 
W/Qualifications 

DEFER/ 

SERVICE~\ 

APPROVAL 

W/Qualifications 

ARMY 

2 

3 

4 

- 

9 

NAVY 

1 

1 

3 

- 

5 

AIR  FORCE 

1 

* 

3 

- 

8 

JOINT 

- 

- 

- 

- 

- 

4 

8 

10 

0 

22 

Four  systems  (18  percent)  had  completed  their  test 
programs  for  the  FSD  phase  successfully  and  met  all 
thresholds;  Phalanx,  Stinger,  ATCA,  and  MLRS.  No 
significant  T&F  issues  were  identified  and  test  cata 
appeared  to  support  an  unqualified  decision  to  proceed 
to  rate  production  ("maturation  phase"  in  the  case  of 
MLRS).  Eight  systems  (36  percent)  received  approval 
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to  proceed  to  rate  production  w :  t  r:  certain  oualiiica- 
t  ions  ;  i.e. ,  proceed i no  to  rate  product : on  war  t ade 
conditional  or  dependent  on  re col  vino  spec; l  r;  p  r  c i  - 
loros ,  usually  in  parallel  with  the  production  effort 
and  oeneral lv  not  requirinu  another  forxal  DSAPC  re- 
fore  proceeding.  Nearly  naif  (45  percent)  of  all  pro¬ 
grams  at  milestone  III  received  approval  for  only 
limited  production.  For  these  programs,  the  risks  of 
committing  to  product  ion  were  regarded  to  he  toe  h  l  a  r. 
without  an  extended  evaluation  of  limited  production 
hardware.  A  favorable  full-scale  production  decision 
was  postponed  until  additional  testing  had  demon¬ 
strated  either  the  successful  correction  of  identified 
deficiencies  or  the  achievement  of  thresholds  which 
had  not  vet  been  tested. 


The  re  were  no  milestone  III  reviews  that  resulted 
ir.  outriqht  cancellation  of  ongoing  programs  or  even 
in  denial  of  proceeding  to  production  in  some  manner. 

' 3 )  Summary 

Overall,  nearly  three-quarters  (74  percent)  of 
the  milestone  II  and  III  reviews  resulted  in  >ss  than 
unconditional  approval:  10  out  of  16  (62  oercent’ 
milestone  IIs,  and  18  out  of  22  (82  percent;  mile¬ 
stone  Ills.  The  totals  for  each  decision  category  are 
shown  by  year  at  the  bottom  of  each  section  of 


ANALYSIS  OF  TRENDS  AND  RELATIONSHIPS 


A  plot  of  the  DS ARC  decisions  is  shown  in  Exhibit 
IV-3.  This  araph  shows  the  total  number  of  DSARC  II  and 
III  reviews  held  each  year  and  the  number  that  resulted  in 
unconditional  approval.  If  the  data  are  normalized,  as 
shown  in  Exhibit  IV-4,  it  can  be  seen  that  the  percentage 
of  unconditional  approvals  is  decreasing  almost  linearly 
from  about  40  percent  in  19_7  to  less  than  15  percent  in 
the  first  3  months  of  198C. 

These  data  could  be  interpreted  as  reflecting  the  in¬ 
creased  emphasis  on  operational  effectiveness  ana  suita¬ 
bility  that  occurred  during  this  time  frame  and,  perhaps, 
a  more  cautious  attitude  dv  defense  dec i s i ^nmake r s  regard- 
ina  deploving  systems  that  had  not  met  tne.r  operational 
suitability  goals.  However,  it  is  also  possible  that  this 
trend  results  from  the  "bow  wave"  phenomena--programming 
for  more  systems  than  can  be  afforded.  A  1977  Defense 
Science  Board  (DSB)  task  force  report  on  the  defense  ac¬ 
quisition  cycle  estimated  that  the  bow  wave  averaged  2'0* 
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Ex::::  i  *■  I7-r  l.-rs  1  r.  jI  phar-e*-  icj  I  or  !»-r  eacr.  ww  pen. 
•:  sr  em  tr.af  had  a  mi  Iwfor.e  :  I  or  III  r«vie’*  bar  i  r.o  hr.o 
under  studv.  In  and  1 t  : 1  we  ,  t  he  CSApr  type  ir.  bat 

r  ;;e  de  „•  i  «  ion  tv:,»>  ,  a  nar  rat  i  vt  :*-r;cr  ir>t  ion  of  T  s,V.  i  3  rues 

-  t  a  i  no  i  in  tne  DSAPO  document  at  ion ,  and  ar,  indication 

anno  rent  r-M.-ron  's'  for  lone  than  unconditional  ap  - 
;  r  ■■.mi  i who applicarlej  are  shown. 

K>:n :  r:  f  I  V-*i  shows  the  f  r^oaenev  •  f  aooa  rent  reasons 
:  r  n  rd  :  t  i  ana  1  decisions  as  derived  f  r  or.  Exit  in  it  IV- S. 

:  1  ■  n  ■' :  there  data  shows  that  failure  to  meet 

•  o  r  -i  I  is  :  ,v  the  predominant ,  apparent  reason  for  corn: : 

•  :  i  .:  torts.  Twenty -one  of  the  18  DSAPhs  wrier  re- 

■-r:  r t he  pace  of  tno  program  or  ordered  additional  7 
;  r  p  O  OV.O  t?,T  failed  tO  T  O  f  C  00^  0  1  0016  Of 

:  * per;'  rmar.ee  or  operational  suitability  thresholds. 
Idris  included  -■  out  of  1C  DSAHC  7  Is  <  5C  percent)  and  16 
Mt  of  16  DSAPC  Ills  (89  percent).  This  would  imply  tna 
the  performance  spec i f icat ions  were  set  unrealistically 
niar.,  that  the  reviews  were  be  i  nc  held  too  early  in  the 
acquisition  eye  1 e  to  allow  for  the  desired  decree  of  sys 
to.Ti  maturity,  or  that  the  DSAPC  failed  to  appreciate  the 
development  a  1  nature  of  the  svsterns  under  review. 

Inadequate  T&E  data  appeared  to  be  a  reason  for  less 
tr.an  unconditional  approval  in  only  nine  cases:  one  USA 
II  and  eiqht  DSAP.C  Ills.  These  included  three  Army  pro¬ 
grams  (the  Poland  and  Patriot  missile  systems  and  the 
fieshting  vehicle  system),  two  Caw  programs  f  the  y.CLWC-  a 
Aeois  weapon  system) ,  and  three  Air  Force  systems  (the 
ALCM ,  the  EF-111A  tactical  jamming  system,  and  the  C-5A 
wing  modification) . 

As  shown  in  the  note  to  Exhibit  IV-6,  the  apparent 
reasons  occurred  sinalv  and  in  combination.  The  most  fr 
•guent  combination  was  a  breach  of  performance  threshold 
and  other  reasons;  next  was  threshold  breach  plus  inade¬ 
quate  T&E  data;  followed  by  threshold  preach  alone  and 
other  reasons  alone.  Inadequate  T&E  data  was  the  sole 
reason  for  conditional  decisions  in  only  two  cases,  the 
C-5A  and  the  fiahtina  vehicle  system  (FVS).  Interest¬ 
ing  lv ,  both  of  these  'and  the  Aegis  review)  resulted  in 
full  production  release  with  qualifications  rather  than 
on.lv  limited  production  authority  q ranted  the  other 
proa  r  ams  . 

As  shown  in  Exhibit  IV- 7,  the  percentage  of  condi¬ 
tional  decisions  that  can  be  attributed,  at  least  in  oar 
to  inadequate  T&F  data  has  risen  steadily  since  191'. 
Inadequate  T&E  data  appeared  to  be  a  reason  for  imposing 
some  form  of  constraint  on  more  than  half  of  the  n roar  am 
which  had  such  decisions  in  the  first  8  months  of  1980. 
However,  it  is  inappropriate  to  extrapolate  this  curve 
into  the  future  since  it  reflects  policies  that  were  in 
effect  5  or  more  years  aao. 
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1  \  MILESTONE  III 

f  I  MILESTONE  II 


SELECTION  OF  PP^CRAMS  FOR  EL’PTHER  ST'.'CV 

On  tne  basis  of  this  initial  screening,  it  was  tec 
o  concentrate  on  those  milestone  III  decisions  that  sp¬ 
eared  to  involve  inadequate  TS.E  data.  Two  programs  were 
elected  from  each  Service  for  further  studv,  as  fallows: 


SYSTEM 


DSARC  III  DATE 


DECISION 


J  Army 

- , 

i  Patriot 

21 

August  1980 

Limited  production. 

i 

j  Roland 

31 

May  1979 

Limited  production 

i  Maw 

1  —  - - * 

1 

j 

Aea  i  s 

19 

January  1978 

Full-scale  production  . 

with  Qualifications  ! 

MCLWG 

1 

8 

November  1977 

j 

Limited  production 

i 

i 

i  Air  Force 

ALCM 

17 

April  1980 

Limited  production 

i 

EF-111A 

12 

December  1978 

! 

Limiter:  Production. 

★ 

★ 

* 

*  * 

Chapter  V 

contains 

a  decision  profil 

e  of  <=ach.  of  the  s e - 

lected  svst 

ems .  In 

order  to  identifv 

the  under! v i n c  causes 

of  inadecruate  T&E  data,  issues  arising  from,  each  milestone 
review  are  analyzed  in  the  context  of  pr og ramspec i f ic 
events  and  of  the  overall  environment  in  whicr.  the  decision 
was  made.  For  this  reason,  it  may  be  helpful  to  unfold 
Appendix  D,  the  diachronic  chart,  for  reference-  during  the 
review  of  the  decision  profiles  contained  in  the  next 
chapter . 
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V.  DECISION  PROFILES 


This  chapter  describes  the  events  surrounding  the 
milestone  III  review  of  the  six  programs  selected  for 
further  study.  In  each  case,  there  is  e  idence  that  the 
DSARC  principals  felt  there  were  inadequate  T&E  data  for 
making  an  unconditional  decision  to  proceed  to  full-scale 
production.  This  chapter  will  provide  insight  on  how  the 
DSARC  integrated  T&E  issues  with  other  factors  in  arriving 
at  their  final  recommendations. 

Each  decision  profile  in  this  chapter  is  dividec  into 
the  following  three  parts: 

.  A  factual  summary  of  the  formal  DSARC 
documentation 

.  A  subjective  interpretation  of  the  DSARC  decision 
based  on  interview  results  and  analysis  of 
program-spec i f ic  and  exogenous  factors 

.  A  historical  schedule  showing  program  milestones 
and  key  events. 

In  addition,  each  decision  can  be  put  in  historical  oer- 
spective  by  referring  to  the  diachronic  chart  contained  in 
Appendix  D.  The  decision  profiles  are  arranged  in  chrono¬ 
logical  order,  starting  with  the  earliest  milestone  III 
review . 

1.  MAJOR  CALIBER  LIGHTWEIGHT  GUN  (MCLWG) 

The  MCLWG  program  was  initiated  in  1963  and  progressed 
under  Navy  management  with  several  major  redirections  in 
scope  and  configuration,  including  changes  in  both  caliber 
and  platform.  A  decision  in  1973  to  use  the  DD-963  class 
as  the  platform  for  the  MCLWG  intensified  development 
efforts  and  could  be  considered  the  actual  start  of  FSD. 
Exhibit  V-l  is  a  summary  of  the  official  documentation 
surrounding  the  milestone  III  review;  Exhibit  V-2  shows 
the  key  events  leading  to  the  DSARC. 

Interest  in  the  MCLWG  program  waxe^  '  waned  through¬ 
out  most  of  its  life.  It  only  came  '  'Dtc  •  t  tent  ion  as  a 
result  of  Congressional  and  GAO  inte  ?i  some  13  years 
after  beginning  development.  It  was  the  kind  of  program 
that  had  neither  high  national  priority  nor  a  complete 
commitment  by  its  proponent  Service.  Although  the  DSARC 
approved  the  Navy's  request  for  limited  production  in 
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EXHIBIT  V- I 

Dec  ision  Document  a  t 1  or. 

Major  Ca 1 1  no r  L 1 g h t we i g n t  Cun  ( MCLWG ) 


TEMP:  TEMP  No.  016,  dated  Septemoer  9,  1974  was  the  basis 

for  testing  up  to  DSARC  III.  This  TEMP  was  not  co¬ 
ordinated  with  OSD.  A  new  TEMP  (No.  118)  was  in 
preparation  at  the  time  of  the  DSARC. 

PRE-DSARC  III  DC P :  DCP  #167,  dated  28  October  1977, 

addressed  the  issue  as  to  whether  production  of  tne 
MCLWG  should  be  approved.  It  acknowledged  that  a 
large  degree  of  concurrency  was  planned  but  stated 
that  ", . .concurrency  is  unavoidable  in  a  program  with 
many  of  the  system  components  still  in  development..." 
and  also  that,  "...due  to  tne  long  production  time 
required  for  the  MCLWG,  it  is  not  feasiole  to  have  tne 
related  development  project  comiDleted  prior  to  com¬ 
mencement  of  MCLWG  production."  Concurrency  risr.s 
were  judged  to  be  manageable.  The  DCP  recommenced 
option  1  (of  five  options  presented,  including  termi¬ 
nation)  which  called  for  commencing  procurement  in 
FV79  and  procuring  32  MCLWGs. 

T&E  ASSESSMENT:  The  T&E  assessment  dated  4  November  1977 
agreed  that  the  gun  and  mount  had  performed  well  in 
tests  hut  since  many  elements  had  beer,  missing,  stated 
that  additional  testing  was  required  to  integrate  the 
total  system  and  demonstrate  overall  system  opera¬ 
tional  capability.  It  concluded  that  "...  MCLWG  will 
be  operationally  effective  if  program  objectives  are 
met  for  improved  ammunition  and  other  systems  now 
under  development."  Consequently,  the  DDT&E  recom¬ 
mended  that  the  Navy  develop  an  integrated  program 
structure  to  ensure  early  testing  of  essential  systems 
prior  to  the  initial  buy  of  five  guns  and  that  a  new 
risk  assessment  of  the  other  systems  under  development 
be  prepared. 

DSARC  DECISION:  The  milestone  III  review  was  held  on  8 
November  1977.  The  resulting  memoranda  from  the 
USDR&E  (2  December  1977)  and  Deputy  Secretary  of 
Defense  (6  January  1978)  authorized  limited  production 
of  the  first  seven  guns  of  a  17  gun  procurement 
(option  1  of  the  DCP).  The  recommendation  for  limited 
rather  than  full  production  was  based  on  the  fact  that 
there  were  "...  concurrent  ammunition  and  related  ship 
system  component  developments  during  the  4  years 
required  for  production  of  the  first  qun. . . .  "  In 
authorizing  the  limited  production,  the  Deputy  Secre¬ 
tary  of  Defense  requested  that  a  program  review  be 
held  in  mid  FY80  before  release  to  full  production 
"...in  time  to  effect  the  FY81  procurement...." 
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November  1977,  the  program  was  cancelled  shortly  tne re- 
after.  The  program  did  not  nave  a  MFNS ,  tut  th.ere  w«:.  a 
Navy  operational  requirement  (OR) .  Mo  DS  ARC  II  was  rcl;. 


The  question  of  inadequate  T&E  data  primarily  :: 
cerned  tne  fact  that  the  MCLWG  was  oeing  testeu  on 
War  II  destoyer  rather  tnan  on  the  new  DD963  on  wr.i 
would  have  been  installed.  This  meant  that  the  gun 
ne  tested  with  an  obsolescent  analog  fire  control,  s 
( MK  6  2 )  instead  of  the  digital  MK86  for  which  it  was 
signed.  in  addition,  the  next  generation  improved 
tionai  munitions  (ICM)  and  semiactive  laser  guided 
tiles  ( SALGP)  which  would  have  given  the  gun  ennanc 
caoaDilitv  were  not  vet  available. 


a  , 
c  r. 


,o  r 


t  •  •  rr. 


ue  - 
c  o  r. 


While  a  TEMP  aid  exist  before  the  major  DT&E  and  IGT&E 
(althouan  just  barely),  it  was  not  approved  by  OSD  or  ever, 
forwarded  there  for  comment.  This  action  was  not  requires 
by  existing  policy  nor  was  the  program  even  designated  as 
a  major  system  at  the  time.  OSD  involvement  grew  out  of  a 
controversy  between  the  Naval  Sea  Systems  Command  'NAVSFA; 
project  personnel  and  the  independent  tester,  Operational 
Test  and  Evaluation  Force  (OPTEVFOR)  . 


DT  was  conducted  on  land-based  f 1971)  and  shipboard 
(1975)  facilities  and  demonstrated  tnat  the  system  per¬ 
formed  functionally  and  that  contract  tecnnical  require¬ 
ments  had  been  met.  Shipboard  test  used  the  DD-943  as  a 
surrogate  for  the  DD-963.  R &M  was  poor  initially,  cut 
improvements  were  made.  OT  III  operational  evaluation 
(OPEVAL)  was  conducted  from  September  1975  to  June  1976 
and  verified  the  DT  results  in  an  operational  environ¬ 
ment.  However,  the  tests  were  limited  m  scope  due  to  cr.e 
lack  of  the  DD-963,  the  new  ammunition,  and  associates 
systems . 

In  order  to  settle  some  earlier  disagreements,  tne 
Chief  of  Naval  Operations  (CNO)  in  a  letter  dated  1  Novem¬ 
ber  1973  set  forth  the  acceptance  criteria  for  opera¬ 
tional  effectiveness  and  suitability  of  the  MCLWG. 
Nevertheless,  the  Commander,  OPTEVFOR  ( COMOPTEVFOR )  con¬ 
cluded  that  the  MCLWG  would  not  oe  operationally  effective 
based  on  the  mission  requirements  stated  in  the  FOR  'not 
including  improved  accuracy  and  effectiveness  that  were 
planned  with  the  SALGP  and  ICM  ammunition) .  Only  after 
CNO  intervention  were  approval  for  service  use  'AST',  and. 
plans  made  for  initial  production.  However,  concurrent 
GAO  studies  tended  to  support  tne  OPTEVFOR  results  shewing 
a  lack  of  accuracy  and  overall  effectiveness.  GAG  recom¬ 
mended  tnat  OSD  review  the  program  before  proceeding  with 
production. 
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At  the  same  time.  Congress,  responding  to  the  per¬ 
ceived  value  of  la rge-ca 1  ibe r  ammunitions  during  Vietnam 
and  the  decommissioning  of  the  Navy's  last  8-inch  cruiser, 
was  exerting  extraordinary  pressure  to  proceed  wit.n,  or 
even  accelerate,  the  program.  All  of  this  attention 
caused  the  program  to  be  elevated  to  formal  DSARC  review 
status.  OSD  insisted  that  the  MCLWG  be  viewed  as  a  system 
to  include  the  MK86  fire  control,  the  new  ammunition,  and 
other  systems  with  which  it  would  eventually  be  used. 

They  directed  that  a  DCP  and  integrated  program  plans  ce 
prepared  and  staffed  and  chat  all  of  the  other  require¬ 
ments  for  a  review  of  a  major  system  be  met.  In  the 
meantime,  OSD  deferred  the  FY77  production  money  'already 
appropriated  by  Congress)  pending  the  outcome  of  the  DSARC. 

In  light  of  the  differences  in  maturity  of  the  MCLWG 
and  its  associated  systems,  the  DSARC  decision  to  proceed 
with  limited  production  and  to  require  a  follow-on  OT&L 
and  DSARC  IIIB  review  of  the  performance  of  the  integrated 
system  would  appear  reasonable.  What  eventually  led  to 
the  program's  demise  was  the  disconnect  between  the  DSARC 
and  the  PPBS. 

Shortly  after  the  DSARC,  the  Navy,  while  construct:",'.: 
POM  80,  realized  that  there  was  insufficient  out  -yea : 
money  to  procure  guns  at  the  rate  just  approved  by  the 
DSARC.  Because  of  OSD’s  attitude  to  force  a  choice  be¬ 
tween  alternatives,  the  program  dropped  out  of  the  Navy 
POM  and,  without  the  out-year  money,  the  FY78  initial  pro¬ 
curement  was  also  cut.  Eventually,  the  entire  program  was 
cancel  led . 

In  the  final  analysis,  the  MCLWG  program,  was  less 
affected  by  inadequate  T&E  data  than  it  was  by  the  lac-:  cf 
a  consensus  regarding  the  need  for  the  program  and  the 
lack  of  harmony  between  the  DSARC  and  PPBS.  Nevertheless, 
the  following  observations  are  warranted: 

.  From  a  TsE  standpoint,  the  decision  not  to  allo¬ 
cate  the  resources  associated  with  the  DD-961  w a 
key.  As  one  former  project  officer  s aid,  "We  go 
an  ODsdete  ship,  a  primitive  fire  control  sys¬ 
tem,  and  ancient  ammo.  " 

.  The  difference  in  maturity  between  the  MCLWG  and 
its  associated  systems  was  clearl-.-  recognized  by 
the  DSARC.  Given  the  long  production  lead  time 
of  the  qun,  the  limited  production  decision  was 
probably  a  rational  response  to  the  inadequate 
data  available. 
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The  controversy  regarding  the  results  of  OTII  is 
not  necessarily  an  indication  of  a  breakdown  in 
the  T&E  process.  Although  strict  adversarial 
relationships  are  seldom  beneficial,  checks  and 
balances  are  a  desirable  feature  that  often  pre¬ 
vent  large-scale  mistakes.  Some  in  the  develop¬ 
ment  community  felt  that  OPTEVFOR  had  exceeded 
its  authority  and  adopted  an  attitude  of  recom¬ 
mending  cancellation  of  any  system  that  failed  to 
meet  its  spec  i  f  icat  ions .  However,  it  may  be  t.nat 
the  misunderstanding  derived  from  a  difference  of 
opinion  on  the  scope  and  objectives  of  the  OPEVAL 
tests . 

.  The  controversy  surrounding  milestone  ill  and  t re 
eventual  cancellation  of  the  program  also  could 
be  attributed  to  the  absence  of  a  milestone  II 
decision  that  would  have  affirmed  the  mission 
need  and  established  meaningful  thresholds  ana 
constraints . 

2.  AEGIS  WEAPON  SYSTEM 

The  Aegis  DSARC  I IA  was  held  in  June  1974  and  directed 
the  Navy  to  "...  effect  simplification  of  the  AEGIS  design 
and  to  demonstrate  Aegis/Standard  Missile  (SM-2)  compati¬ 
bility."  Three  and  one-half  years  later,  in  January  1978, 
OSD  approved  production  of  :he  system  in  spite  of  substan¬ 
tial  evidence  of  inadequate  T&E  data.  A  summary  of  the 
formal  documentation  surrounding  the  DSARC  is  shown  in 
Exhibit  V- 3 ;  a  historical  schedule  is  shown  in  Exhibit  V-4. 

Shipbuilding  is  a  special  case  in  the  acquisition  pro¬ 
cess.  It  has  more  in  common  with  the  construction  indus¬ 
try  than  it  does  with  the  development  and  production  of 
most  other  types  of  weapon  systems.  The  long  lead  times, 
large  capital  investment,  complexity  of  integration  with 
associated  systems,  and  the  relatively  small  number  of 
units  produced  (compared  to  missiles  or  even  airplanes! 
blurs  the  distinction  between  R &D  and  production  and  maxes 
the  term  "limited  production"  meaningless. 

Aegis  is  no  exception.  Considering  that  the  Aegis 
weapon  system  and  the  CG-47  shipbuilding  program  were 
merged  for  “-he  milestone  III  review,  it  was  an  enormously 
complex  pi  im  whose  total  cost  was  over  $15  billion. 

One  would  •  .^ect  that  a  program  of  this  size  would  be  con¬ 
troversial.  In  fact,  it  was  during  its  early  development 
'nut-  t  n  r-  vjavy  was  havinq  internal  arguments  about  wnich 
:  j-rorm  to  out  Aegis  on.  However,  after  the  sinking  of 
r  !•  .  :  pint  ny  a  Russian-made  antiship  missile  in 
•  •  ■  ■  w  :  .  consensus  on  the  urgent  need  for  an 


EXHIBIT  V- 3 

Decision  Documentation 
Aegis  Weapon  System 


TEMP:  TEMP  #124  was  published  in  April  1976  and  approved 

within  the  Wavy  by  OP03.  It  was  sent  to  DDT&E  out 
there  was  no  formal  review  or  approval.  A  revised 
TEMP  #124 (Rev.  1)  was  prepared  prior  to  DSARC  III. 

PRE-DSARC  III  DCP :  DCP  #16  (Rev  2),  dated  18  November 

1977,  concluded  that  “...technical  risk  for  the  AEGIS 
Weapon  System  is  low,"  that  "...extensive  Development 
Test  and  Evaluation  (DT&E)  and  Operational  Test  and 
Evaluation  (OT&E)  of  Aegis  in  USS  MORTON  SOUND  have 
demonstrated  that  the  system  has  no  residual  risxc," 
and  that  the  integration  risks  are  related  to 
"...computer  program  completion."  The  DCP  concluded 
that  "there  were  no  T&E  issues  which  offer  high  risk, 
and  recommended:  "Proceed  with  the  Aegis  Weacon  S'”- 
tem  Acquisition  Plan  commencina  in  FY78  and  structure 
to  support  the  Aegis  shipbuilding  program." 

T&E  ASSESSMENT:  The  T&E  assessment  dated  17  January 

1973  stated  that  Aegis  had  completed  considerable  DT& 
and  IOT&E  and  that  OT  ITIA  had  demonstrated  potential 
over  existing  systems  in  spite  of  the  absence  of  k«v 
software,  ,  and  ORTS  elements.  Software  require¬ 
ments  were  judged  to  pose  moderate  technical  nut  high 
schedule  r’s<s.  The  DDT&E  also  concluded  that  al¬ 
though  "...OT  to  date  is  inadequate  to  assess  the 
effectiveness  and  suitability  of  the  Aegis  Weapon  Svs 
tem  in  a  realistic  threat  environment,"  that  fcllov-c 
testino  would  add  confidence.  He  recommended  that 
production  of  one  Aegis  and  installation  on  the  DDG-4 
were  warranted.  Follow-on  production  would  be  su-'ec 
to  the  results  of  additional  DT  and  OT  of  EDM- 3  at 
CSEDS  in  Morestown,  N.J. 

PA&E  ASSESSMENT:  In  a  memo  dated  13  December  1977,  ar.e 
ASD'PA&E)  raised  two  issues  for  DSARC  consider  at  i  or. : 
(1;  use  of  the  Army's  Patroit  missile  instead  of  the 
SM-2,  and  (2)  effective  operation  of  the  s'/ste-  in  ?. 
h i g n- 1 h rea t  environment . 

DSARC  DECISION:  The  milestone  ill  review  war  hell  nr,  19 
January  1978.  The  resulting  Deputy  St  -retary  of 
Defense  memo,  sin"!  2  March  1978,  aut  loriced  the  w.av 
to  proceed  with  production  of  the  Aeq i s  and  installa¬ 
tion  on  the  DDG-4 7  as  recommended  subject  only  to  sub 
mission  of  specified  planning  and  cost  data  to  OSD. 
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improved  antiair  warfare  capability  foi  u.S.  Naval  surface 
forces.  Consequently,  Aegis  enjoyed  a  priority  within  the 
Navy  second  only  to  TRIDENT. 

Aegis  was  begun  under  the  total  package  procurement 
( TPP)  concept  and  converted  to  the  DSARC/mi lestone  review 
system  shortly  after  its  inception  in  1969.  Planning  for 
the  DT/OT  tests  in  support  of  the  milestone  III  review 
centered  around  an  agreement  in  1976  between  the  DDT&E  and 
the  Navy  regarding  sequential  test  phases  designated  IIIA, 
IIIB,  etc.,  and  incremental  DSARC  reviews.  It  was  at  this 
point  the  DDT&E  acknowledged  that,  subject  to  the  concur¬ 
rence  of  COMOPTEVFOR,  the  successful  completion  cf  IOT&E 
on  the  USS  NORTON  SOUND  would  provide  sufficient  justifi¬ 
cation  for  release  of  the  lead  DDG-47  and  first  production 
of  the  Aegis  system. 

project  personnel,  OPTEVFOR,  and  the  technical  commu¬ 
nity  ail  participated  in  drafting  the  TEMP  which  imple¬ 
mented  this  agreement.  It  covered  in  detail  the  objec¬ 
tives  of  OTIIIA  and  the  high-level  parameters  that  would 
be  demonstrated.  It  called  for  operation  and  maintenance 
by  service  personnel  when  appropriate  and  it  made  reason¬ 
able  tradeoffs  with  regard  to  realistic  threat  scenarios. 
The  TEMP  was  then  viewed  as  the  controlling  planning  docu¬ 
ment  on  T&E  matters.  "It  was  the  Bible,"  said  one 
ind 1 v id  ua  1 . 

And,  according  to  some,  it  was  a  model  TEMP.  It  was 
one  of  the  first  to  be  prepared  in  accordance  with  the  May 
1976  revision  of  DODI  5000.3  (and  the  subsequent  revision 
to  OPNAVINST  3960.10),  and  it  was  held  as  an  example  for 
otner  weapon  systems. 

Testing  of  the  Aegis  system  was  conducted  on  an  engi¬ 
neering  development  model  (EDM-1),  a  single  quadrant  early 
version  of  the  SPY-1  radar,  installed  in  the  USS  NORTON 
SOUND.  Through  December  1976,  28  missile  (SM-1)  firings 
were  conducted  by  Navy  crews;  21  were  successful,  7  were 
attributed  to  missile  failures.  DT  IIIA  of  EDM-1  SM-1  was 
concluded  in  May  1977  and  consisted  of  nine  SM-2  at-sea 
firings  (six  successes  and  one  partial  success).  OT  IIIA 
was  conducted  between  June  and  July  1977  witn  20  days  at 
sea  at  the  Pacific  Missile  Range.  The  system  was  evalu¬ 
ated  in  numerous  nonfirinq  exercises,  and  there  were  seven 
SM-2  firings  during  which  the  system  was  operated  and 
maintained  by  the  USS  NORTON  SOUND  crew,  except  when 
specific  external  maintenance  assurance  was  authorized  ana 
doc  umen  ted . 

The  DT  and  OTIIIA  tests  were  mostly  successful  and 
both  COMOPTEVFOF  and  DDT&E  concluded  that  the  substance  of 
the  1976  agreement  had  oeen  met.  Residual  concerns  were 


that,  in  reality,  only  a  small  portion  of  the  entire  Aegis 
weapon  system  had  been  tested  and  that  numerous  questions 
remained  unanswered.  Missing  pieces  included  full 
coverage  from  the  SPY-1  radar,  key  software,  elements  of 
the  command  and  control  system,  and  the  operational 
readiness  tests  system  (ORTS) .  In  addition,  DDT&E 
observed  that  in  spite  of  the  progress,  testing  to  date 
was  inadequate  to  assess  the  system  in  a  realistic  threat 
envi ronment . 

In  light  of  all  these  inadequacies,  it  may  be  sur¬ 
prising  that  the  DSARC  recommended  that  Aegis  proceed  into 
full  production.  There  are  several  explanat ions : 

.  Project  personnel  argued  that  the  tests  conducted 
demonstrated  successful  operation  of  the  high 
risk  elements  of  the  system;  that  the  body  of 
data  was  consistent  with  the  plan  to  move  incre¬ 
mentally  to  the  next  phase. 

Early  agreement  with  OSD  was  key.  The  ground 
rules  for  proceeding  into  production  had  been 
negotiated  well  in  advance. 

.  The  program  had  a  high  national  priority.  Fur¬ 
thermore,  there  had  been  years  of  consensus 
building  within  the  Department  of  Defense  via 
interagency  briefings  and  reviews.  One  person 
remarked  that  it  was  inconceivable  that  the  DSARC 
would  do  anything  other  than  recommend  full 
production. 

.  The  program  had  maintained  relatively  low  pro¬ 
file;  i.e.,  there  had  not  been  major  overruns  or 
other  adverse  publicity.  The  GAO  was  only  rou¬ 
tinely  critical  and  there  was  strong  Congres¬ 
sional  support. 

.  The  emphasis  on  demonstrated  operational  suita¬ 
bility  had  not  yet  taxen  hold. 

.  The  nature  of  ship  acquisition  made  full  produc¬ 
tion  essentially  the  same  as  limited  production, 
which  might  have  been  more  likely  for  other  Kinds 
of  weapon  systems  given  the  unknowns  that  still 
ex i s ted . 

Interestingly,  the  major  opposition  to  production  of 
the  Aegis  came  from  the  ASD(PA&E)  who  was  oiven  specific 
responsibility  within  OSD  for  OT&E  shortly  after  this 
DSARC.  PA&E  was  concerned  that  there  was  insufficient 
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evidence  that  Aegis  would  function  effectively  in  a  high- 
threat  environment.  He  also  wanted  to  know  why  the  Navy 
could  not  use  the  Army  Patriot  missile  rather  than  the 
Navy  SM-2. 

3 .  EF-111A  TACTICAL  JAMMING  SYSTEM 

The  EF-111A  DSARC  II  met  in  October  1974.  The  result¬ 
ing  Deputy  Secretary  of  Defense  memorandum  of  23  January 
1975  approved  a  38-month  FSD  program  (instead  of  the 
faster  paced  28-month  program  proposed  by  the  Air  Force). 
It  is  also  stated  that  there  were  important  questions 
regarding  cost  and  operational  feasibility  that  could  only 
be  answered  by  testing  and  that  the  Air  Force  should  "... 
complete  the  test  program  set  forth  in  the  DCP  with  satis¬ 
factory  results  prior  to  the  DSARC  III  review."  An  up¬ 
dated  DCP  (No.  126B)  was  issued  on  13  May  1975.  A  summary 
of  the  decision  documentation  surrounding  the  milestone 
III  review  is  shown  in  Exhibit  V-5;  a  historical  schedule 
is  shown  in  Exhibit  V-6. 

The  EF-IIIA  milestone  III  decision  is  a  study  in  con¬ 
trasts.  The  Air  Force  was  convinced  that,  while  they  were 
forced  to  cut  back  on  testing  due  to  overruns  in  R$,D,  the 
DT/OT  they  did  conduct  was  adequate,  that  all  performance 
thresholds  stated  in  the  DCP  were  met,  and  that  their  con¬ 
tract  with  DDT&E  (via  the  TEMP)  had  been  satisfied.  On 
the  other  hand,  DDT&E  and  the  GAO  were  just  as  convinced 
that  development  test  and  evaluation  were  incomplete  and 
that  operational  testing  was  too  limited  to  provide  a 
realistic  threat  environment  for  evaluating  operational 
effectiveness  and  suitability.  Speci f ically ,  they  noted 
that  there  were  209  technical/design  deficiencies  reported 
by  the  Air  Force  Test  and  Evaluation  Center  (AFTEC)  of 
which  perhaps  40-50  were  duplications.  The  majority  of 
these  were  in  the  operability,  reliability,  and  maintain¬ 
ability  area.  In  addition,  the  GAO  in  particular  remained 
skeptical  of  the  EF-lllA's  military  worth  or  cost- 
ef  feet i veness . 

From  a  historical  perspective  it  appears  that  OSD,  in 
fact,  did  break  their  contract  with  the  Air  Force.  Re  1  i  - 
anility  and  maintainability  began  to  emerge  as  important 
themes  in  mid-1978,  after  the  TEMP  had  been  reviewed  and 
approved.  By  this  time,  the  ASD  (MRA&L)  and  his  staff 
were  emphasizing  R&M;  some  believe  that  they  dominated  the 
DSARC  decisionmaking  process  during  this  period.  Their 
concern  regarding  the  lack  of  "blue  suit"  maintenance  may 
have  been  justified,  but  the  intention  to  use  contractor 
maintenance  for  IOT&E  was  clearly  specified  in  the  TEMP. 
The  indications  are  that  this  milestone  decision  was  at 
the  leading  edge  of  new  policy  initiatives  and  that  it 
became  the  £est  case  for  those  in  OSD  that  wanted  to  set 
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EXHIBIT  V- 5 

Decision  Documentation 
EF-111A  Tactical  Jamming  System 


TEMP:  A  revised  TEMP  covering  plans  for  the  combined 

DT&E/ IOT& E  was  published  on  15  September  1977.  It  was 
reviewed  and  approved  by  DDT&E  prior  to  the  start  of 
testing,  although  not  early  enough  to  significantly 
alter  test  resources  or  plans. 

PRE-DSARC  III  DCP :  DCP  #126C  (draft)  issued 

1  November  1978  stated  that  "...all  DT&E/IOT&E  planned 
objectives  were  met  during  the  Phase  IB  combined  test 
program.  It  is  the  consensus  of  these  organizations 
[SPO,  TAC,  and  AFTEC]  that  the  testing  was  as  compre¬ 
hensive  as  existing  U.S.  test  resources  would  support 
and  that  additional  testing  is  not  required..."  It 
also  stated  that  all  performance  thresholds  of  the  13 
May  1975  DCP  were  met  and  requested  authority  to  pro¬ 
ceed  with  rate  production  of  42  aircraft  and  parallel  1 
correction  and  demonstration  of  deficiencies. 

I 

T&E  ASSESSMENT:  The  T&E  assessment  dated  1  November  197S  i 
stated  that  there  was  a  lack  of  adequate  test  assets 
and  facilities  to  assess  performance  under  certain  ; 

conditions;  that  there  were  overall  and  specific  main-  i 
tainability  and  reliability  problems,  including  elec-  ' 
tromagnetic  interference  (EMI),  software,  inadequate 
support  equipment  (and  others),  and  limited  evidence 
of  "blue  suit"  maintenance  participation ;  and  that  | 

"...a  favorable  production  decision  cannot  be  sup¬ 
ported  until  (these)  operational  suitability  problems  J 
have  been  corrected  and  postively  demonstrated..." 

DSARC  DECISION:  The  milestone  III  review  was  held  on 

12  December  1978.  The  resulting  Deputy  Secretary  of 
Defense  memo  dated  12  February  1979  concluded  that  j 

IOT&E  was  generally  favorable,  but  the  "...assessment  j 
of  the  operational  suitability  results  disclosed  a 
number  of  problems  in  reliability  and  maintainabil¬ 
ity..."  It  directed  a  phased  production  program  with 
a  parallel  suitability  evaluation  "...to  serve  as  a 
precaution  against  future  increases  in  cost  which 
might  arise  from  the  need  to  redesign  and  retrofit 
unreliable  equipment..."  The  Air  Force  was  directed 
to  provide  reports  for  OSD  review  at  3-month  intervals 
with  a  recommendation  for  program  continuation  or 
termination.  Based  on  these  reports,  the  Acquisition 
Executive  would  determine  the  advisability  of  full- 
scale  production. 
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an  example.  Furthermore,  the  EF-111A  was  the  first  major 
program  to  have  a  DSARC  III  in  nearly  a  year.  Some 
believe  this  gave  OSD  staffers  extra  time  to  investigate 
and  prosecute  their  emerging  belief s--that ,  in  effect, 
they  were  ready  and  waiting  for  the  EF-111A  review.  One 
OSD  official  said,  "There  were  new  cooks  and  a  clean 
kitchen.  When  the  meat  was  put  on  the  table,  everyone  dug 


It  appears  that  the  GAO  also  adopted  the  EF-lllA  as  a 
special  case.  They  had  just  completed  several  reports  on 
operational  testing  and,  in  particular,  reported  in 
December  1977  that  "Air  Force  DT&E  needs  improvement." 

From  June  1978  to  August  1980,  there  were  five  separate 
GAO  reports  on  the  EF-lllA--roughly  one  every  6  months. 

The  decision  to  limit  production  ani  conduct  addi¬ 
tional  T&E  was  more  a  product  of  the  times  than  of  the 
previously  established  way  of  doing  business.  Subsequent 
policy  changes  have  emphasized  R&M  issues  and  implemented 
procedural  changes  that  would  have  facilitated  the  EF-illA 
milestone  review.  These  changes  notwithstanding,  the 
limited  production  decision  was  a  controversial  one. 
Project  personnel  argued  that  it  unnecessarily  added 
2  years  to  the  program  and  increased  total  program  cost  by 
$100-$200  million.  In  addition,  some  feel  that  stretching 
out  the  production  from  3  to  4  years  was  motivated  by 
fiscal  constraints  rather  than  by  a  risk  assessment  of 
technical  problems. 

Adequacy  of  T&E  data  was  clearly  an  issue  during  the 
decisionmaking  process.  It  was  highlighted  in  the  DCP, 
primarily  as  a  result  of  the  GAO  claim  that  "...  accom¬ 
plished  testing  was  inadequate  to  proceed  with  a  DSARC  III 
decision."  Other  observations  on  events  leading  to  the 
review  include: 

.  R&D  problems,  including  a  $15-million  overrun, 
were  tne  primary  cause  of  the  reduction  in  the 
number  of  test  aircraft  (from  two  to  one),  the 
combined  DT/OT,  and  the  reliance  on  contractor 
maintenance. 

.  The  test  program  was  conducted  essentially  as 

specified  in  the  TEMP;  however,  the  program  was 
overtaken  by  changes  in  policy  that  altered  the 
expectations  of  the  DSARC  principals  and  left  the 
Air  Force  largely  unprepared  for  the  review 
process . 

.  The  Air  Force  felt  the  deficiencies  found  in 

IOT&E  could  be  corrected  by  engineering  change 
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proposals  (ECPs)  during  full-scale  production. 

OSL'  insisted  on  a  more  cautious  f  lx-and-retest 
app  roach . 

.  The  DDTi E  found  that  a  lacK  of  adequate  test 

assets  and  test  facilities  prevented  an  assess¬ 
ment  of  tne  EF-lllA's  ability  to  operate  in  cer¬ 
tain  environments.  GAO  agreed  ana  recommended 
that  the  DOD  improve  its  capability  for  testing 
electronic  warfare  systems  and  undertaxe  a  dedi¬ 
cated  effectiveness  test  when  improved  threat 
radar  simulators  were  in  place. 

.  The  DSARC  was  informed  of  the  cost  of  delaying 
the  EF-IllA  program  in  order  to  conduct  addi¬ 
tional  T&E,  but  did  not  share  the  Air  Force's 
sense  of  urgency  for  the  system.  It  optea  for  ar. 
approach  that  would  reduce  the  risx  of  subsequent 
redesign  and  retrofit. 

4.  ROLAND  MISSILE  SYSTEM 


Full-scale  development  on  Roland  oegan  as  a  result  of 
a  DSARC  II  held  in  February  1974.  Subsequently,  the  Army 
evaluated  three  foreign  systems  that  were  under  develop¬ 
ment  as  well  as  improvements  to  the  u.S.  Chaparral  sys¬ 
tem.  in  January  1975  the  Deputy  Secretary  of  Defense 
approved  the  Army's  source  selection  recommendation  to 
acquire  the  Franco- German  Roland  II.  Since  tnat  time, 
there  were  two  major  redirections  in  the  programs:  (1)  as 

a  result  of  cost  growth,  a  special  DSARC  held  in 
September  1976  restructured  the  program  to  reduce  the  num¬ 
ber  of  missile  firings  from  146  to  100,  and  (2)  again  in 
response  to  cost  growth,  a  special  program  review  by  the 
DSARC  principals  in  June  1978  restructured  the  program, 
released  FY78  IPF  and  other  funds,  and  requested  an 
increase  of  testing  at  the  performance  boundaries. 

In  spite  of  significant  doubts  as  to  the  reliability 
and  maintainability  of  the  Roland  missile  system,  limited 
production  was  approved  in  May  1979.  The  decision  docu¬ 
mentation  is  summarized  in  Exhibit  V-7;  a  historical 
schedule  is  shown  in  Exhibit  V-8. 

The  Roland  program  is  the  only  one  of  the  six  selected 
for  further  study  that  had  a  heavy  multinational  flavor. 

In  fact,  the  terms  DT  and  OT  were  never  formally  used  m 
deference  to  the  phrase,  technology  transfer,  which  was 
supposed  to  reflect  the  maturity  of  the  system  and  the 
commonality  of  U.S.  Roland  with  its  Franco-German  counter¬ 
part.  Undoubtedly,  the  North  Atlantic  Treaty  Organization 
(NATO)  influence  added  another  dimension  to  the  milestone 
review  and  further  complicated  the  question  of  T&F, 
adequacy . 
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EXHIBIT  V-  7 

Decision  Documentation 
Short  Range  Air  Defense  (SHORAD)  System 
US  ROLAND 


TEMP:  A  formal  TEMP  was  not  available  during  the  FSD 

phase,  although  a  draft  TEMP  was  submitted  just  prior 
to  DSARC  III  in  April  1979.  The  US  Roland  test  pro¬ 
gram  was  based  on  the  test  philosophy  contained  in  the 
April  1976  "Powers  Report”  (see  text). 

PRE  DSARC  III  DC P :  DCP  #95,  Rev.  C  dated  22  March  1979, 
"For  Comment  Draft"  addressed  the  issue  of  whether 
Roland  was  ready  to  enter  the  production  phase.  The 
DCP  did  not  include  a  specific  recommendation  although j 
six  program  alternatives  (from  termination  to  accel-  ; 
eration)  were  described.  I 

T&E  ASSESSMENT:  In  his  memorandum  dated  30  May  1979,  the  j 
DDT&E  concluded  that  testing  "...had  verified  that 
European  technology  has  been  successfully  transferred 
and  that  Roland  has  essentially  met..."  the  system 
performance  parameters  contained  in  the  Army  ROC.  In 
add i t ion  : 

i 

.  That  a  significant  effort  would  be  required 
to  achieve  the  RAM  goals 

.  That  the  system  exhibited  susceptibility  to  ! 
downlink  jamming  and  additional  testing  was 
required  to  verify  performance  j 

.  That  adverse  weather  testing  conducted  was  j 
inadequate 

.  That  maintainability  was  not  tested. 

Consequently,  the  DDT&E  recommended  that  any  produc¬ 
tion  beyond  low-rate  production  should  be  contingent 
on  demonstrated  RAM  growth,  and  that  modified  test 
plans  and  results  of  planned  tests  be  forwarded  to  OSD 
for  review.  In  a  subsequent  memo  dated  1  June  1979, 
the  DDTSE  further  stated:  "Testing  in  the  areas  of 
ECM  susceptibility,  maintainability,  adverse  weather 
performance  and  warhead  lethality  was  either  incom¬ 
plete  or  has  yet  to  be  accomplished." 


EXHIBIT  V-  7  (continued) 


DSARC  DECISION:  The  DSARC  convened  on  31  May  1979  and 

determined  that  the  Roland  program  should  be  pursued 
and  that  the  Army  should  be  authorized  to  enter  lim¬ 
ited  rate  production.  in  his  memorandum  to  the 
SecDef /DepSecDef  dated  4  June  1979,  the  USDRE  also 
stated:  "When  the  results  of  currently  planned  reli¬ 

ability  tests  are  available,  we  will  conduct  a 
DSARC  IIIB  review  to  determine  the  appropriate  time  to 
enter  full-scale  production."  The  SecDef  decision 
memorandum  dated  6  June  1979  stated  that  "Roland 
acquisition  cost  estimates,  current  progress,  and  per¬ 
formance  are  satisfactory  for  limited  production,"  and 
granted  approval  for  low  rate  initial  production 
(LRIP)  in  FY79  and  FY80.  It  also  directed  the  Army  to 
provide  information  on  proposed  support  concepts  to 
ASD  (MRA&L)  within  120  days  and  a  revised  TEMP  to 
DDT&E  within  45  days.  Finally,  it  called  for  a 
DSARC  IIIB  review  of  RAM  capability  before  entry  into 
full  production. 


EXHIBIT  V-£ 
Key  Events 


A  TEMP  was  not  prepared  or  submitted  for  OSD  review 
until  just  prior  to  the  DSARC  III  Decause,  according  to  PM 
personnel,  "TEMPS  weren't  required  until  early  1979." 
(Actually,  the  revision  of  DOD  Directive  5000, 
quired  submission  of  a  TEMP  to  OSD  for 
in  April  1978.)  Fortunately,  there 
nation  on  an  equivalent  document,  the 
test  plan  (CTP) .  One  member  of  the 
in  bed  with  us  since  1976  and  the  OSD  staffers  were  active 
participants  in  our  Test  Integration  Working 
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review  was  issued 
was  very  early  coordi- 
Army  coordinated 
PMO  said,  "DDT&E  was 
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This  high  degree  of  coordination  stemmec  from  an  ear¬ 
lier  review  of  Roland  testing,  which  was  headed  by  MG 
Powers,  then  Commander  of  the  Test  and  Evaluation  Command 
(TECOM) ,  and  subsequently  referred  to  as  the  "Fowers 
Report."  In  1975  the  Roland  program  was  subject  to  severe 
funding  pu r t u r ba t ions  which  upset  the  existing  T&E  plans. 
The  purpose  of  the  Powers  report  was  to  restructure  the 
test  program  in  light  of  the  reduced  funding  and  to  effect 
efficiencies  in  the  test  process.  It  essentially  laid  cr.~ 
around  rules  for  T&E  through  milestone  III.  Even  though 
it  substantially  reduced  the  amount  of  testing  to  he  con¬ 
ducted,  it  acted  as  an  implied  contract  Detween  OSD  ana 
the  Service.  At  the  time  of  adoption,  the  schedule  for 
the  test  program  was  acknowledged  to  be  "...  optimistic 
and  of  high  risk,  but  the  data  produced  will  permit  a 
moderate  to  low  risk  production  decision."  Further,  tne 
report  noted  "...  a  major  deviation  from  the  standard 
acquisition  process  in  that  the  production  decision  will 
have  been  made  without  an  adequate  evaluation  of  RAM, 
training,  anc  logistics  support."  As  :re  PM  official  put 
it,  "We  knew  this  was  the  least  we  w*  re  am r.q  tc  get  away 
with..." 


The  flight  test  program  consisted  of  107  missile  fir¬ 
ings  of  which  70  were  without  failures,  31  had  failures  of 
some  sort,  and  6  were  declared  no  tests.  The  flight  test 
program  identified  quality  control  and  design  deficiencies 
which,  according  to  the  DCP,  were  subsequently  resolved. 

In  addition,  a  nonflignt  test  program  was  conducted  which 
identified  design  shortfalls.  These  included  electronic 
countermeasures,  reliability,  and  human  factors.  The 
independent  evaluation  of  the  operational  effectiveness 
and  suitability  of  US  Roland  conducted  fcv  the  T.S.  Army 
Operational  Test  and  Evaluation  Agency  (OTEA)  concluded 
that  the  system  had  demonstrated  capability  to  perform 
all-weather  mission  although  reliability  was  insufficient 
and  maintainability  had  not  been  tested.  OTEA  concluded 
that  system  effectiveness  and  reliability  should  re  fur¬ 
ther  examined  in  follow-on  evaluation. 

However,  tne  program  was  subject  to  other  forces 
ing  its  development.  One  of  those  came  with  the  c r a r  a >- 
administrations  in  J ar.ua rv  197  7  when  Dr.  »<•  .  rv  ret  3  -• 


Dr.  Currie  as  the  USDRE.  Previously,  the  manaaement 
philosophy,  at  least  as  perceived  by  the  Project  Manage¬ 
ment  Office  (PMO) ,  was,  "We  know  you  have  problems,  but 
get  it  in  the  field  and  we'll  fix  it  later."  Later,  that 
changed  to  "Fix  it  and  prove  you've  fixed  it  before  we 
talk  about  production."  In  addition,  there  was  pressure 
from  the  GAO  and  Congress  to  provide  satisfactory  evidence 
of  Roland's  performance  before  commi 1 1 i ng  . to  production. 
Some  even  wanted  to  revisit  the  whole  selection  of  Roland 
as  the  Army's  future  all-weather  short-range  air  defense 
system. 

OTEA  and  DDT&E  agreed  that  the  system's  shortfalls 
included  R&M  and  ECM  and  that  adverse  weather  and  main¬ 
tainability  testing  was  inadequate.  GAO  raised  similar 
objections  but  DOD  responded  by  saying  that  the  GAO  report 
was  particularly  misleading  about  the  adequacy  and  amount 
of  testing  to  be  completed  prior  to  the  production 
decision. 

The  R&M  issue  was  perhaps  the  most  serious.  m^e 
Roland  DSARC  followed  by  less  tnan  30  days  the  DS,  ~>C  III 
on  the  XM-1  tank  which  had  serious  reliability  and  main¬ 
tainability  problems.  This  criticism  was  also  levied  at 
Roland.  It  appears  that  Roland  became  another  victim  of 
the  generally  critical  atmosphere  that  began  with  the 
EF-111A  (in  December  1978).  The  DDT&E  had  concluded  that 
failure  to  demonstrate  reliability  goals  was  a  chronic 
problem  and  that  it  required  a  change  in  the  way  DOD  was 
doing  business.  One  official  said  "Once  the  gun  had  seen 
cocked  and  fired,  it  was  easier  to  use." 

Cost  was  another  issue  which  may  have  impacted  the 
DSARC  decision  to  limit  production.  The  priority  of  air 
defense  had  softened  somewhat  since  the  Roland  program  was 
started  and  Patriot  was  consuming  the  majority  of  what  was 
left  in  the  FY80  budget  and  the  FY81  POM.  One  theory  is 
that  out-year  cost  increases  due  to  uneconomic  production 
buys  were  accepted  in  order  to  meet  current  year  dollar 
constraints;  i.e.,  that  the  T&E  issues  were  a  convenient 
way  of  addressing  the  budgetary  pressures  associated  with 
the  " bow  wave . " 

Finally,  while  the  Army  may  have  been  willing  to  trade 
Rolands  for  Patriots,  the  proponents  of  RSI  and  the  "two- 
way  street"  observed  that  Roland  was  the  only  foreign  sys¬ 
tem  being  considered  for  production  by  the  United  States. 
One  of  the  specific  issues  identified  for  the  DSARC  was, 
"What  are  the  RSI  implications  of  not  procuring  Roland?" 

The  Roland  program  may  or  may  not  be  typical,  but  it 
provides  an  interesting  vignette;  (1)  test  planning  was 
coordinated  well  in  advance;  but  (2)  cost  and  schedule 


pressures  forced  the  Army  to  conduct  limited  IQT&E  on  P*D 
hardware  not  representative  of  the  production  configura¬ 
tion;  further,  (3)  new  philosophies  regarding  decision 
criteria  emerged  after  the  original  plan;  this  resulted  m 
(4)  inadequate  T&E  data  to  support  a  production  decision; 
however,  (5)  OSD  decided  to  go  ahead  with  limited  produc¬ 
tion  due  to  other  factors. 

5.  AIR-LAUNCHED  CRUISE  MISSILE  ( ALCM) 

The  ALCM  program  was  initiated  in  July  1973,  and  a 
DSARC  I  was  held  in  February  1974  which  approved  entry 
into  an  18-month  advanced  development  phase.  In  December 
1974,  a  DSARC  II  review  resulted  in  disapproval  of  entry 
in  the  full-scale  development  phase.  A  subsequent  DSARC 
IA  review  in  March  1975  resulted  in  approval  of  an  ad¬ 
vanced  development  demonstration  for  proof -of -concept . 
During  the  period  FY75  through  the  first  quarter  of  FY77, 
the  Air  Force  and  the  prime  contractor,  Boeing,  conducted 
tests  intended  to  demonstrate  system  integration  in  a  fly¬ 
ing  vehicle  within  established  threshold  performance 
parameters.  Subsequently,  a  DSARC  II  review  was  held  in 
January  1977  which  approved  the  Boeing  design  (AGM-86)  for 
FSD ,  The  DSARC  also  directed  the  establishment  of  a  Joint 
Cruise  Missile  Project  Office.  Nine  months  later,  in 
September  1977,  the  USDRE  reoriented  the  program  by  di¬ 
recting  a  competitive  development  between  the  Boeing 
AGM-86  and  the  General  Dynamics  AGM-109  (an  air-launched 
version  of  their  BGM-109  sea-launched  cruise  missile).  In 
February  1980,  the  Air  Force  selected  the  AGM-86B  config¬ 
uration  for  production  and  deployment. 

The  air-launched  cruise  missile  program  became  the 
major  moderization  effort  for  the  strategic  bomber  force 
when  the  B-l  was  cancelled  by  President  Carter  in  mid- 
1977.  The  new  administration's  Under  Secretary  of  Defense 
for  Research  and  Engineering,  Dr.  William  Perry,  was  a 
strong  proponent  of  the  technology  used  in  ALCM  and  other 
members  of  the  cruise  missile  family,  and  he  provided  a 
focal  point  for  support  of  the  program  within  OSD.  Per¬ 
haps  because  of  this  support  and  because  ALCM  was  a  criti¬ 
cal  part  of  the  strategic  force  planning,  it  was  approved 
for  limited  production  in  April  1980  in  spite  of  a  large 
number  of  test  issues  which  were  inconclusive  or  undeter¬ 
mined.  The  DSARC  documentation  is  shown  in  Exhioit  V-9 
and  the  historical  schedule  in  Exhibit  V-10. 

ALCM  was  an  ambitious,  success-oriented  program  with  a 
high  degree  of  planned  concurrency.  IOT&E  was  to  be  con¬ 
ducted  on  immature  and  incomplete  systems.  For  example, 
integration  into  the  B-52  aircraft  was  to  be  accomplished 
after  the  production  decision.  Combined  DT&E/IQT&E  tests 
were  conducted  with  developmental  prototypes  procured  from 
competing  contractors. 


EXHIBIT  V-9 

Decision  Documentation 
Air  Launched  Cruise  Missile  ( ALCM ) 


TEMP:  The  28  March  1979  ALCM  TEMP  was  the  baseline 

document  for  combined  DT&E/IOT&E  although  tests  were 
already  underway  by  the  time  of  approval.  A  draft 
revised  TEMP  was  provided  to  DDT&E  in  anticipation  of 
the  DSARC  III  review. 

PRE  DSARC  III  DCP/IPS:  DCP  #136  dated  3  March  1980  sup¬ 
ported  a  milestone  III  decision  by  recommending 
"...that  the  ALCM  be  approved  for  full  production  as 
outlined  in  this  DCP  with  the  provision  that  the  FOT&E 
program  be  closely  monitored  to  assure  that  perfor¬ 
mance  and  reliability  thresholds  are  achieved." 

Because  the  ALCM  Source  Selection  was  not  completed  at 
the  time,  the  DCP  included  demonstrated  performance 
parameters  but  did  not  discuss  specific  results  of 
tests  obtained  during  the  competition.  Threshold 
achievement  and  other  data  were  to  be  reported  at  or 
before  DSARC  III.  According  to  the  DCP,  the  major 
issue  was  the  risk  associated  with  the  reliability/ 
maintainability  of  the  system.  The  DCP  also  stated 
that  "...follow-on  development  and  operational  test 
and  evaluation  will  adequately  address  remaining 
issues  of  risk  and  concern." 

T&E  ASSESSMENT:  The  DDT&E  submitted  his  independent  T&E 
assessment  in  a  memorandum  dated  11  April  1980.  He 
stated  that  significant  data  were  obtained  on  most  of 
the  key  performance  parameters  but  that  some  desired 
tests  were  "...  impacted  by  the  limited  number  of 
flights  (four  were  shortened  by  malfunctions)  and 
somewhat  hindered  by  instrumentation  problems."  The 
DDT&E  reported  that  a  substantial  amount  of  DT  and  OT 
remained  to  be  done  because  of  subsystems  still  under¬ 
going  development  (radar  altimeter,  B-52  OAS ,  and  sup¬ 
port  equipment).  Although  the  Air  Force's  proposed 
combined  DT&E/FOT&E  emphasized  these  areas,  the  DDT&E 
concluded  that  the  DT  and  OT  test  results  to  date 
indicated  that  the  "...ALCM  air  vehicle  is  potentially 
capable  of  meeting  operational  requirements."  How¬ 
ever,  he  also  cautioned  that  "...more  extensive  and 
realistic  operational  testing  with  production  repre¬ 
sentative  missiles,  carrier  aircraft,  and  support 
equipment  is  essential  to  successful  production  and 
deployment  of  ALCM."  Finally,  he  urged  timely  comple¬ 
tion  of  the  remaining  development  and  early  opera¬ 
tional  testing  of  those  systems.  He  suggested  subse¬ 
quent,  periodic  OSD  reviews  of  follow-on  test  results. 
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DSARC  DECISION:  The  DSARC  III  was  held  on  17  April  1980. 
The  resulting  SDDM,  dated  30  April  1980,  approved  pro¬ 
duction  leading  to  a  minimum  capacity  to  produce  40 
missiles  per  month,  established  reliability  and  main¬ 
tainability  thresholds  and  goals,  and  directed  the  Air 
Force  to  take  certain  actions  as  recommended  by  the 
DSARC.  The  significant  T&E  related  directives 
included : 

.  Formal  submission  of  an  updated  TEMP  witnin 
60  days  that  defines  a  program  for  determ¬ 
ining  key  ALCM  parameters  for  which  limited 
test  data  exist  and  for  evaluating  systems 
still  under  development 

.  Provision  of  reports  to  OSD  in  October  1980 
and  March  1981  prepared  by  AFTEC  concerning 
progress  made  in  the  test  program 

.  Continuation  of  ram  efforts  and  report  to 
the  Cruise  Missile  Executive  Committee. 
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The  test  program  involved  ground  testing  of  the  inte¬ 
grated  air  vehicles,  including  evaluation  of  radar  cross 
section,  infrared,  electromagnetic  pulse  (EMP) ,  (IR)  sig¬ 
nature,  (EMI/electromagnetic  compatibility  (EMC),  engine/ 
air  frame  compatibility,  engine  performance,  aerodynamic 
performance  (wind  tunnel  tests) ,  wa r head/impact  fuze  tests 
(sled  tests),  and  maintainability).  The  tests  were  con¬ 
ducted  on  ten  flights  each  of  the  AGM-86B  and  the 
AGM-109.  The  Air  Force  Flight  Test  Center  (AFFTC)  was 
responsible  for  the  DT  portion  of  the  tests  and  AFTEC  was 
responsible  for  the  OT  portion.  Each  test  flight  was 
designed  to  accomplish  multiple  DT  and  OT  objective^. 
According  to  the  DCP,  "USAF  personnel  conducted  'ha-:.'- 
maintenance  on  the  competing  systems  as  part  of  the  over- 
nil  eval uation. " 

The  pace  of  the  test  program--20  flights  within  an 
18-month  period--stressed  both  the  developer  and  the 
tester.  Furthermore,  there  were  neither  management  „e- 
serves  nor  opportunities  to  recover  gracefully  from  tech¬ 
nical  problems  as  indicated  by  test  failures  if  they 
occurred-. 

In  such  a  program,  the  TEMP  becomes  a  critical  docu¬ 
ment  for  communicating  the  ground  rules  to  all  concerned 
and  ensuring  there  are  no  T&E  surprises  at  the  milestone 
review.  It  appears  that  the  ALCM  TEMP  did  just  that. 

Once  negotiated,  it  was  viewed  by  the  Air  Force  as  a 
contract  with  OSD  and,  to  the  extent  possible,  it  was 
strictly  adhered  to. 

However,  shortage  of  missiles,  test  failures,  range 
limitation,  competetive  prototyping  (which  diluted  the  T&E 
effort),  and  the  absence  of  subsystems  still  undergoing 
development  resulted  in  a  body  of  T&E  data  that  was,  in 
the  end,  judged  inadequate  to  make  a  full  production  deci¬ 
sion.  AFTEC’s  independent  evaluation  cited  eight  perfor¬ 
mance  parameters  that  were  inconclusive  or  undetermined 
due  to  a  need  for  more  testing.  These  parameters  in¬ 
cluded:  overall  operational  effectiveness,  terminal 

accuracy,  terrain  following,  and  launch  envelope  (for 
which  definite  estimates  of  performance  could  not  be  made 
because  of  the  lack  of  test  points  and  insufficient  data) ; 
mission  planning,  software  suitability,  weapon  system  com¬ 
patibility,  and  interoperability  (because  critical  aspects 
of  the  system  were  not  available  for  test).  Three  other 
parameters  were  judged  deficient  and  13  satisf actory . 

The  DDT&E  concluded  that  more  extensive  and  realistic 
tescing  was  necessary.  While  there  was  some  disagreement 
between  AFTEC  and  the  SPO  on  the  interpretation  of  the 
IOT&E  results,  it  was  clear  that  there  was  still  much 
uncertainty  in  the  operational  effectiveness  and  suit¬ 
ability  of  the  system. 


Nonetheless,  the  DSARC  recommended  that  the  Air  Force 
proceed  with  limited  production.  In  this  case,  it  appears 
that  the  national  priority  and  urjency  for  the  program 
coupled  with  high-level  DOD  support  tipped  the  balance  in 
favor  of  accelerated  development.  Also,  the  decision  to 
go  ahead  with  the  ALCM  was  made  in  conjunction  with  the 
decision  to  cancel  the  B-l  bomber,  which  was  formally 
announced  3  months  later.  Furthermore,  the  reviewers  knew 
in  advance  that  the  production  decision  would  have  to  be 
made  with  less  than  full  data  and  they  accepted  the  asso¬ 
ciated  risks. 

6.  PATRIOT  MISSILE  SYSTEM 


The  Patriot  weapon  system  development  program  was 
initiated  in  1965.  Contractor  tests  and  simulations  were 
conducted  from  1967  to  1972  to  demonstrate  the  Patriot 
surveillance  and  guidance  principles  and  evaluate  hardware 
fabricated  to  initial  design  specifications.  FSD  began  in 
March  1972  when  the  DepSecDef  approved  DCP  #50.  The 
development  was  redirected  in  1974  when  the  DepSecDef 
instructed  the  Army  to  provide  early  flight  verification 
of  the  track  via  missile  (TVM)  guidance  principle  while 
exercising  fiscal  restraint  in  other  areas.  A  DSARC 
review  was  held  in  January  1976  in  which  the  decision  was 
made  to  resume  full-scale  engineering  development.  Since 
that  time,  prototype  fire  units  were  produced,  significant 
design  changes  were  made  (particularly  the  conversion  to 
digital  modules  in  the  missile  guidance  system)  and  a 
total  of  50  contractor  flight  tests  were  conducted.  As 
shown  in  the  decision  documentation,  Exhibit  V-ll,  the 
DSARC  agreed  to  limited  production  of  the  system  in  August 
1930  even  though  there  was  strong  evidence  of  inadequate 
T&E  data.  A  historical  schedule  is  shown  in  Exhibit  V-12. 

Patriot  gained  a  reputation  as  the  longest  development 
program  in  the  history  of  the  Army,  if  not  the  entire 
DOD.  With  the  total  development  time  approaching  20 
years,  there  was  enormous  pressure  on  the  PM  and  the  Army 
to  go  to  the  DSARC  as  soon  as  possible  in  order  to  transi¬ 
tion  into  the  production  phase  and  start  putting  something 
into  the  field.  Ironically,  this  pressure,  emphasized  by 
the  possible  loss  of  funding  or  threats  of  cancellation, 
in  itself  contributed  to  the  fact  that  inadequate  T&E  data 
were  available  to  the  decisionmaking  process.  As  the 
milestone  review  neared,  the  Army  decided  to  conduct  OT 
prior  to  DT  so  that. they  would  at  least  have  partial  DT 
and  OT  results  to  present  to  the  DSARC.  Unfortunately, 
this  meant  that  things  that  should  have  been  discovered 
and  corrected  in  DT  were  not;  and  that,  consequently,  the 
system  was  not  sufficiently  mature  to  have  a  meaninqful 
IOT&E .  A  senior  OTEA  representative  said,  "The  tests  were 
mostly  negative,  some  positive  and  largely  inconclusive." 


EXHIBIT  V-ll 
Decision  Documentation 
Patroit 


TEMP:  A  CTP  was  prepared  early  in  the  FSD  program  and 

coordinated  among  the  Army  test  community;  however, 
there  was  never  formal  OSD  approval.  In  preparation 
for  DSARC  III,  the  Army  submitted  the  CTP  dated  August 
1978  which  was  rejected  as  having  too  much  detail  for 
a  management  review.  A  TEMP  submitted  in  April  1979 
was  rejected  because  it  did  not  have  enough  deta'. 
regarding  post-DSARC  III  plans.  A  revised  TEMP,  da 
11  July  1980,  was  subsequently  approved. 

PRE-DSARC  III  DCP :  The  Patroit  Air  Defense  System  DCP, 

dated  20  August  1980,  described  three  alternatives  for 
pursuing  the  production  and  deployment  of  the  system 
(#l-remain  in  FSD,  #2-accelerate  the  program,  and  43- 
initiate  a  system  confirmation  program)  plus  one 
alternative  to  terminate  the  program.  On  3  July  1980, 
the  ASARC  adopted  alternative  #2  as  the  preferred 
approach;  however,  the  DCP  states:  "...to  alleviate 
reliability  and  investment  risks  while  maintaining  an 
accelerated  program,  execution  of  the  newly  introduced 
alternative  #3  is  recommended.  This  alternative  pro¬ 
vides  a  prudent,  yet  quick,  program  to  expend  USAREUR 
air  defense  capabilities  while  reducing  attendant 
risks  and  keeping  program  costs  relatively  low."  In 
addition  to  the  program  alternatives,  the  DCP  ad¬ 
dressed  four  DSARC  issues,  including  the  impact  of  not 
testing  certain  elements  of  the  system,  the  adequacy 
of  existing  test  data,  and  the  risks  associated  with 
the  software  development  plan.*  In  each  case,  the  DCP 
advances  a  positive  case  for  the  Patroit  system.  The 
DCP  was  supplemented  by  an  Integrated  Program  Summary 
(IPS),  dated  14  July  1980. 

T&E  ASSESSMENT:  The  DDT&E  in  his  assessment  dated  15 

August  1980  concluded  that  "...the  Patroit  system,  as 
tested,  did  not  demonstrate  its  readiness  for  opera- 


Others  included  the  number  of  ready  missiles  per  fire 
unit,  the  performance  of  TVM  in  an  ECM  environment, 
and  the  electronic  counter-countermeasures  (ECCM) 
capability  to  meet  the  growing  threat. 
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tional  employment."  Problems  included  missile  reli¬ 
ability  (particularly  in  an  operational  environment), 
inadequate  ECCM  under  some  conditions,  poor  fire  unit 
reliability,  unsatisfactory  mobility,  and  immature  and 
constantly  changing  software.  In  addition,  because 
the  system  was  maintained  by  the  contractor  during  01' 
using  nonstandard  corrective  maintenance  techniques, 
the  "...present  data  in  fault  localization  and  repair 
(were)  of  little  use  ..."  and  maintainability  could 
not  be  determined.  In  the  event  of  a  decision  tc  pro¬ 
ceed  at  a  minimum  production  rate  (perhaps  anticipat¬ 
ing  this  outcome) ,  the  DDT&E  recommended  an  extensive 
structured  program  of  tests  and  reviews:  "The  test 
and  review  program  should  be  established  to  insure 
that  substantial  progress  toward  achieving  satisfac¬ 
tory  Patroit  operational  suitability  is  demonstrated 
before  production  is  permitted  to  rise  above  the  mini¬ 
mum  level. " 

DSARC  DECISION:  DSARC  III  was  held  on  21  August  1980. 

The  resulting  SDDM  to  the  Secretary  of  the  Army,  dated 
10  September  1980,  took  exception  to  the  Army's 
request  but  approved  "...limited  production  at  a  rate 
appropriate  for  confirmation  of  the  design  as  manufac¬ 
tured  on  production  tooling." 

The  Secretary  of  Defense's  position  was  based  on  the 
fact  that  "...performance  in  ECM  conditions  has  not 
been  fully  demonstrated  by  operational  and  development 
testing,  nor  has  there  been  demonstration  of  accept¬ 
able  levels  of  reliability  and  maintainability  in 
either  the  fire  units  (FU)  or  in  the  missile."  He 
further  stated  that  "...  hardware  off  production  tool¬ 
ing  in  some  quantity  will  be  required  in  order  to  both 
prove  the  production  design  and  to  provide  enough 
^auipment  for  obtaining  soldier  experience,  and  that 
“.he  limited  production  minimizes  the  financial  risk 
:nc:rr°d  since  anticipated  changes  will  not  result  in 
»pp reel  able  retrofit  cost." 

’  sc  stating,  the  SecDef  established  a  comprehensive 
“et  of  conditions  that  had  to  be  met  before  release  of 
sun  sequent  increments  of  funding.  He  further  required 
submission  of  a  revised  TEMP,  which  followed  the 
Guidelines  establisned  in  the  DDT&E  assessment  of  15 
August  1980,  within  45  days  and  periodic  reviews  to 
determine  the  level  of  production  justified  by  test 
results.  He  even  directed  that  an  award  fee  contract 
based  on  technical  performance  be  used  for  the 
follow-on  effort. 
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The  pm  staff  believes  that  if  they  had  had  another 
year,  they  would  have  had  a  smooth  DSARC  review,  but  tnat 
they  could  not  afford  to  "get  out  line."  That  is,  the 
lead  times  in  the  budget  (PPBS)  process  are  such  that  if 
tney  had  not  had  the  DSARC  when  they  did  and,  in  fact, 
gotten  approval  for  some  form  of  production  they  woulci 
have  lost' the  FY81  money  and  perhaps  risked  termination  of 
the  the  program.  They  also  believe  that  the  noted  defi¬ 
ciencies  in  software,  diagnostics,  and  R&M  could  have  beer, 
fixed  in  parallel  with  a  full  production  program.  OSD 
d i sag  reed  . 

From  the  PM ' s  perspective,  each  of  the  OSD  elements 
acted  as  an  antogonist--pursuing  narrow  technical  in¬ 
terests  without  attempting  to  integrate  all  the  factors 
that  impacted  the  cost  of  delay  versus  the  cost  of  pro¬ 
ceeding.  However,  it  is  also  true  that  the  contract  be¬ 
tween  OSD  and  th,_  Army  regarding  the  adequacy  of  T&E  data 
necessary  to  support  a  production  decision  never  existed. 
Although  the  Army  had  prepared,  staffed  and  was  using  a 
CTP ,  it  was  not  until  gust  before  milestone  ill  that  a 
TEMP  was  approved  by  DDT&E .  This  was  caused  partly  by  the 
fact  that  Patriot  was  always  one  generation  behind  the 
management  philosophy  of  the  day;  i.e.,  the  program  had 
been  overtaxen  by  so  many  changes  in  policies  and  proce¬ 
dures,  not  to  mention  personalities,  that  it  was  always  in 
isolation  some  policy  and,  consequently,  became  fair  game 
for  any  number  of  critics. 

There  were  also  indications  of  lack  of  cooperation 
within  the  Army.  The  PMO  felt  that  TECOM  dwelled  on 
" . . .  the  20  percent  that  went  wrong  rather  than  the  SO 
percent  that  went  right,"  and  that  this  bias  carried  over 
to  the  independent  evaluators,  the  Army  Materiel  Systems 
Analysis  Activity,  AMSAA  and  OTEA.  OTEA  felt  that  much  of 
the  DT  testing  was  not  representative  of  an  operational 
situation  and  refused  to  accept  and  use  some  of  the  data. 
They  also  felt  that  the  system  was  fundamentally  not  ready 
for  OT  and  it  may  be  that  they  conducted  the  test  under 
mild  duress.  This  internal  strife  also  existed  at  the 
Developmental  and  Readiness  Command  (DARCOM)  and  Army- 
Headquarters  was  probably  perceived  by  the  OSD  staff,  each 
of  whom  had  his  own  suggestions  on  how  the  program  ought 
t o  r  e  f  l  xed . 

In  the  end,  however,  the  rising  costs  associated  with 
continued  development  mitigated  the  inadequacy  of  the  data 
and  resulted  in  a  limited  production  compromise. 


SUMMARY  OF  FINDINGS 


The  six  proararrs  studied  varied  widely  w  i  t  r.  r° 
the  specifics  or  their  test  and  evaluation  or,'.:  t  r.e 
to  which  perceived  inadequacy  influenced  the  event 
DSARC  reccrrrrenda  t  ion.  Eac.n  program  and  mi  left-:  ne 
was  very  much  a  product  of  its  special  c  i  rcumntar.c 
the  context  of  the  existing  dec  i  s  i  on  it  ax  :  no  ,-r.v  i  ror. 
However,  as  shown  in  Exhibit  7-13,  so  ire  coxiror,  i  ac 
appa  rent  : 


Three  of  the  programs  were  extremely  lone 
more  years  between  the  start  of  AO  and  r h 
stone  III  review.  Lone  programs  tend  to 
come  by  chances  in  acquisition  and  TSE  pc 
new  management  philosophies  of  < e y  dec  is  i 
makers,  and  even  the  threat. 

Three  of  the  programs  had  attracted  a  iur 
amount  of  attention  from  t.-.e  GAG  and/or  C 
and  had  achieved  a  degree  of  notoriety  wi 
military  or  in  the  punlic  media  as  "had  p 
formers"  in  one  way  or  another.  'Jnd«r  tr. 
circumstances,  it  may  have  become  easy,  e 
fashionable,  to  he  overly  critical  of  to e 
proqr ams . 


Four  of  the  programs  were  still  wrest  1 
fundamental  mission  need:  or  concept  se 
issues  at  milestone  III.  It  is  not  cl 
these  were  legitimate  unresolved  quest 
were  compounded  by  inadequate  TSE  data 
tr.ey  were  spurious  issues  which  simply 
the  decisionmakino  orocess . 


Thro*3  of  toe  proarams  showed  evidence  cf 
service  disputes  over  priority  or  di sag  re- 
regarding  tr.e  adequacy  and  meaning  of  ICT 


Finally,  four  of  the  programs  did  not  hav< 
approved  TEMP  early  enough  in  the  FSD  p.ka: 
sign i f i cant ly  effect  the  adequacy  of  T&E 
presented  to  the  DSARC.  In  fact,  all  cf 
programs  heaan  FSD  long  refer-'1  the  TEMP  w, 
eotar.l  ished  requirement.  However ,  at  lea: 
case,  the  absence  cf  a  TEMP  was  nv°rc 
e  other  agreement  which  served  a  s  im  i  1. 


-one 
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Common  Factors  in  Programs  Studied 
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More  specifically,  there  were  a  number  of  underlying 
causes  for  inadequate  T&E  data  which  could  be  identified 
for  each  program,  as  shown  in  Exhibit  V-14.  These 
included : 


Changes  in  policy 
.  Shifts  in  emphasis 

.  Ambiguities  in  system  scope 

.  R&D  problems 

.  Test  problems 

Each  of  these  is  discussed  below. 

( 1)  Changes  in  Policy 

A  change  in  policy  is  a  substantive  change  in  th 

acquisition  process  as  reflected  by  DOD  directives  or 
instructions  which  occurred  since  a  program's  last 
major  milestone  review.  Included  are  such  thinas  as 
the  objectives,  activities,  and  guidelines  for  acqui¬ 
sition  phases;  the  suporting  documentation  and  proce¬ 
dures;  and  the  criteria  for  moving  from  one  phase  to 
the  next.  As  was  pointed  out  in  Chapter  III,  policy 
swings  can  be  rather  dramatic  and  acquisition  strate¬ 
gies  formulated  at  one  point  in  time  can  be  grossly 
unacceptable  at  another.  This  was  the  case  in  five 
out  of  the  six  programs  studied.  Of  course,  this 
problem  is  exacerbated  by  long  development  programs. 

( 2 )  Shifts  in  Emphasis 

Shift  in  emphasis  is  similar  in  that  it  may  have 
the  same  effect  as  a  change  in  policy;  however,  it  ma 
or  may  not  become  official.  It  deals  with  the  chang¬ 
ing  preoccupation  of  defense  decisionmakers  with  a 
selected  aspect  of  the  acquisition  process,  such  as 
des ign- to-cost ,  or  operational  suitability.  Shifts  i 
emphasis  may  result  from  the  zeal  of  individual  OSD 
staffers  or  from  a  heightened  awareness  of  some  par¬ 
ticular  issue  within  the  acquisition  community.  Both 
the  EF-111A  and  the  Roland  programs  seem  to  have  got¬ 
ten  caught  by  OSD  emphasis  that  was  different  than 
when  the  programs  were  last  reviewed. 

( 3 )  Ambiguities  in  System  Scope 


Inadequate  T&E  data  frequently  resulted  from 
confusion  or  ambiguities  regarding  the  scope  of  the 
system  tested,  i.e.,  how  representative  the  test  hard 
ware  was  of  the  complete  production  configuration 
operating  in  a  realistic  environment  with  all  its 
associated  systems .  There  were  numerous  disagreement 
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EXHIBIT  V- 1  3 

Causes  of  Inadequate  TSE  Data 


1 


at  milestone  III  over  the  "reasonableness"  of  extra¬ 
polating  what  was  actually  tested  to  what  was  planned 
to  be  produced.  This  included: 

.  The  ease  or  difficulty  in  moving  from  a  par¬ 
tial  to  a  full  capability,  most  notably  in¬ 
volving  software  and  system  integration. 

.  The  real;sm  of  the  test  environment,  includ¬ 
ing  operation  and  maintenance  by  typically 
skilled  personnel. 

.  The  use  of  simulation  as  a  substitute  for 
missing  components  or  systems. 

Furthermore,  confusion  over  the  scope  of  the  sys¬ 
tem  to  be  tested  may  have  reflected  a  situation  in 
which  the  critical  test  issues  were  not  well  defined 
and,  consequently,  the  extent  and  nature  of  the  test¬ 
ing  required  was  indeterminable. 

It  is  clear  that  a  fixed  policy  regarding  the 
scope  of  systems  to  be  tested  could  not  have  been 
applied  across  the  board  to  all  the  programs  studied 
because  of  differences  in  technical  risks,  priority, 
etc.  However,  questions  regarding  adequacy  of  T&E 
data  due  to  system  scope  could  have  been  reduced  if 
the  T&E  program  needed  to  address  the  critical  test 
issues  had  been  defined  and  agreed  to  in  advance  on  a 
case-by-case  basis.  A  TEMP  would  have  provided  a 
vehicle  for  such  negotiation  had  it  existed  early 
enough  to  affect  the  test  program. 

( 4)  R&D  Problems 

R&D  problems  encountered  during  development,  as 
part  of  the  normal  R&D  process,  sometimes  consumed 
funds  that  were  planned  for  T&E.  The  resulting  cost 
constraints  frequently  led  to  an  inadequate  number  of 
prototypes  for  training  and  testing  and,  sometimes,  a 
prototype  configuration  that  differed  significantly 
from  that  intended  for  production  (see  system  scope, 
above) .  R&D  problems  also  caused  slips  which  reduced 
the  time  available  for  T&E  prior  to  a  fixed  milestone 
review.  Milestone  III  in  particular  is  sometimes 
viewed  as  fixed  in  time  because  of  an  institutional¬ 
ized  perception  that  appropriated  funds  cannot  or 
should  not  be  released  until  after  DSARC  approval. 
Since  funding  profiles  are  established  years  in  ad¬ 
vance  as  part  of  the  PPBS  process,  and  are  often  based 
on  optimistic  cost  and  schedule  estimates,  it  is  not 
uncommon  for  RDT&E  money  to  be  declining  and  produc¬ 
tion  money  to  be  available  prior  to  completion  of  the 
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full-scale  development  phase.  This  puts  an  added 
pressure  on  the  Service  and  OSD  to  hold  the  DSARC  i ; , 
spite  of  major  deficiencies  (as  in  the  case  of 
Patriot,  for  example)  because  of  concern  over  losing 
the  production  funding. 

The  resulting  schedule  pressures  led  to  inade¬ 
quate  test  time,  short  cuts  in  the  evaluation  of  reli¬ 
ability  and  maintainability,  and  in  logistics  support- 
ability.  It  was  not  uncommon  to  have  the  T&E  time 
planned  in  the  original  schedule  compressed  as  the 
DSARC  date  approached.  OT  in  particular  was  subject 
to  compromise  by  R&D  problems.  Operational  tests  were 
reduced  or  omitted,  OT  was  combined  with  DT  or,  in  the 
worse  case,  hardware  was  released  for  OT  before  all 
developmental  problems  were  resolved. 

Fixing  the  milestone  reviews  as  a  result  of  lead 
time  in  the  budget  process  is  a  mixed  blessing;  it 
sets  aggressive  goals  (which  may  work  to  overcome 
Parkinson's  law);  however,  it  also  encourages  short¬ 
cuts  in  order  to  meet  artificial  deadlines.  Addi¬ 
tional  funding  flexibility  during  the  R&D  to  produc¬ 
tion  transition  may  be  required  to  deal  with  this 
d  i  lemma . 

( 5 )  Test  Problems 


Test  problems  which  prevented  acquiring  adequate 
T&E  data  included  the  availability  of  prototypes  for 
test,  the  availability  of  targets  anc  other  consum- 
aoles,  and  problems  with  range  facilities  and  instru¬ 
mentation.  Interestingly,  range  priority  was  not 
found  to  be  a  problem,  probably  because  the  major  sys¬ 
tem  studied  all  enjoyed  a  higher  priority  than  the 
non-major  systems  with  which  they  had  to  compete. 

A  related  problem  was  that  competitive  prototyp¬ 
ing,  as  in  the  ALCM  program,  diluted  the  T&E  effort  to 
the  point  where  marginal  data  was  available  for  re¬ 
view.  This  is  a  serious  problem  which  must  be  evalu¬ 
ated  in  terms  of  the  whole  question  of  the  value  of 
competition  in  defense  acquisition. 

While  cases  of  inadequate  data  an  be  attributed  to  the 
underlying  causes  described  above,  it  also  appears  that 
the  problem  of  inadequate  T&E  data  is  often  dominated  bv 
milestone  reviews  by  other  decision  drivers  and  that 
limited  production  is  frequently  the  resultino  compro¬ 
mise.  There  is  circumstantial  evidence  that  adequacy  of 
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T&E  sometimes  is  used  as  a  convenient  surrogate  for  short 
term  affordability  problems.  At  other  times  substantive 
question  about  risxs  and  readiness  for  production  are 
overlooKed  in  defer-  ence  to  urgency,  momentum,  national 
priority  and  other  issues  such  as  NATO  RSI. 

★  ★  ★  ★  it 

The  decision  profiles  in  this  chapter  showed  tnat 
while  specific  circumstances  varied,  the  programs  had 
similar  attributes,  there  were  common  underlyinq  causes  o 
inadequate  T&E  data,  and  that  frequently  the  milestone 
decision  was  dominated  by  other,  non-T&E  related,  fac¬ 
tors.  The  next  chapter  integrates  these  and  earlier  find 
ings  into  a  set  of  conclusions  and  recommendations  which 
address  the  study  objectives. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


During  the  period  under  study,  the  majority  of 
milestone  II  and  III  decisions  were  made  conditionally, 
and  the  frequency  of  conditional  decisions  has  increased 
steadily  since  1977.  Use  of  "limited  production"  at 
milestone  III  is  now  commonplace  and  indicates  a  movement 
toward  a  five-milestone  system  as  a  means  of  incremental 
confidence  building. 

Conditional  decisions  can  add  to  the  time  and  cost 
required  to  develop  new  weapon  systems;  however,  they  can 
also  prevent  large-scale  mistakes,  for  example,  by  re¬ 
stricting  capital  investments  until  risks  have  been  mini¬ 
mized.  There  is  no  such  thing  as  the  "correct"  balance 
between  caution  and  exped iency- - i t  changes  as  the  needs 
and  the  resources  on  the  defense  establishment  change. 
Similarly,  it  is  futile  to  search  for  an  absolute  defini¬ 
tion  of  adequate  T&E  data.  It  is  more  constructive  to 
attempt  to  understand  the  way  in  which  T&E  data  are  inte¬ 
grated  into  the  defense  decisionmaking  process  and  to  make 
recommendations  for  improving  that  process. 

1.  CONCLUSIONS 

The  following  conclusions  are  drawn  in  response  to  the 
auestions  that  were  originally  posed  in  Chapter  1. 

( 1 )  Inadequate  T&E  Data  Contribute  to  Some  Condi- 
tional  Decisions 


Inadequate  T&E  data  as  perceived  by  the  DSARC 
principals  are  one  reason  for  conditional  decisions. 
However,  instances  of  inadequate  T&E  data  occur  rela¬ 
tively  infrequently  and  usually  in  combination  with 
otner  reasons,  such  as  failure  to  meet  a  performance 
threshold.  In  any  case,  milestone  III  decisions  are 
commonly  dominated  by  other  factors  such  as  the  ur¬ 
gency  and  national  priority  of  the  program,  the  momen¬ 
tum  of  the  program,  the  long  lead  times  associated 
with  production,  and  the  desire  to  preserve  budgeted 
or  appropriated  funds.  Furthermore,  the  adequacy  of 
T&E  data  appears  sometimes  to  be  used  as  a  convenient 
substitute  for  near-term  affordability  problems. 

Thus,  inadequate  T&E  data  cannot  be  singled  out 
as  the  primary  cause  for  the  conditional  decisions 
studied.  Nevertheless,  it  would  be  desirable  to 
eliminate  individual  instances  where  the  DSARC  imposes 
conditions  which  extend  the  acquisition  cycle  because 
they  auestion  the  adequacy  of  the  T&E  data  presented 
to  them. 
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(  2)  Underlying  Causes  of  Inadequate  T&E  Data  Are  Both 

Internal  and  External  to  the  Program 

Adequacy  of  T&E  data  is  a  difficult  concept  to 
define  precisely.  However,  it  is  clear  that  it  is  a 
function  of  the  expectations  of  the  DSARC  principals 
and  that  these  expectations  are  ill  defined  and  con¬ 
stantly  changing  in  response  to  program-unique  factors 
and  external  events.  Further,  the  principals  them¬ 
selves  change  from  review  to  review  of  a  given  program 
because  the  R&D  cycle  is,  in  general,  longer  than  the 
political  cycle.  This  makes  establishing  a  stable 
framework  for  conducting  individual  program  reviews 
difficult  and  opens  the  larger  issue  of  how  to  insti¬ 
tutionalize  centralized  decisionmaking  in  a  decen¬ 
tralized  operating  environment. 

When  the  DSARC  concludes  that  there  are  inade¬ 
quate  T&E  data  on  which  to  base  an  unconditional  deci¬ 
sion,  it  is  generally  as  a  result  of  a  lack  of  commu¬ 
nication  between  the  Service  and  OSD.  Either  the  ex¬ 
pectations  of  the  principals  were  never  established  or 
there  was  a  change  in  the  program  or  the  decision  en¬ 
vironment  which  was  not  communicated  to  the  other 
party.  In  the  six  cases  studied,  the  underlying 
causes  of  inadequate  T&E  data  most  frequently  included 

.  Changes  in  acquisition  policy  since  the  last 
milestone  review 

.  Shifts  in  emphasis  by  key  players  or  groups 

.  Ambiguities  regarding  the  scope  of  the 

system  to  be  tested,  including 
identification  of  the  critical  test  issues 

.  R&D  problems  which  reduced  the  time  and/or 
funds  available  for  T&E  prior  to  a  "fixed" 
milestone  review  which  is  perceived  as 
necessary  to  preserve  follow-on  funding 

.  Problems  in  conducting  the  tests  themselves 
due  to  difficulty  in  obtaining  test  re¬ 
sources  or  to  range/instrumentation 
limitations . 

Only  the  last  two  are  even  partially  within  the 
control  of  the  Program  Manager  or  developing  Service. 
The  others  primarily  relate  to  the  specific  expecta¬ 
tions  of  the  DSARC  principals  regarding  the  quantity 
and  quality  T&E  data  to  be  presented  at  milestone 
reviews . 
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(3) 


There  Is  Not  Always  Effective  Communication  Be¬ 
tween  OSD  and  the  Services  With  Respect  to  the 
Quantity  and  Quality  of  T&E  Data  That  Is  Required 
for  Milestone  Decisions 


Formal  communication  between  OSL  .nd  the  Services 
regarding  T&E  did  not  routinely  exist  ;ntil  about  1979 
when  the  TEMP  began  to  be  applied  rigorously  as  a 
means  of  establishing  a  "contract"  as  to  the  quantity 
and  quality  of  T&E  required  to  gain  approval  to  pro¬ 
ceed  to  the  next  acquisition  phase.  In  a  sense,  ac¬ 
cording  to  present  policy,  an  approved  TEMP  declares 
and  affirms  on  a  case-by-case  basis  the  expectations 
of  the  DSARC  principals  in  an  attempt  to  bring  sta¬ 
bility  to  the  decisionmaking  process.  However,  it  is 
not  yet  clear  that  future  decisionmakers  will  (or 
should)  be  bound  by  such  agreements  as  program  and 
policy  users  change  and  new  facts  are  brought  to 
light.  Thus,  as  important  as  the  concept  of  the  TEMP 
as  a  contract,  is  its  use  to  provide  cont  muina  com¬ 
munication  between  the  parties  reaarding  test  plans 
that  realistically  reflect  the  acquisition  strategy -- 
including  the  scope  of  the  system  to  be  tested,  the 
DSARC  acceptance  criteria,  and  the  existing  funding 
profile.  Since  1  to  2  years  will  be  required  for  pro¬ 
grams  to  respond  to  the  policies  begun  in  1979,  it  is 
not  yet  known  whether  they  will  be  effective  in  avoid¬ 
ing  the  T&E  inadequacies  that  were  observed. 

( 4 )  The  Way  in  Which  Adequacy  of  T&E  Is  Addressed  in 
the  Present  Milestone  Review  Process  Needs  Im¬ 
provement 

The  independent  T&E  assessment  of  the  DDT&E  is  an 
effective  means  of  evaluating  the  results  as  well  as 
limitations  of  tests  conducted  prior  to  a  milestone 
review.  However,  it  is  but  one  part  of  the  overall 
evaluation  that  must  occur  in  reaching  a  milestone 
decision.  A  judgment  regarding  the  adequacy  of  T&E 
data  cannot  be  made  out  of  context  of  a  total  risk 
assessment  which  considers  the  costs  of  delay,  the 
military  urgency,  and  an  assessment  of  the  conse¬ 
quences  of  success  or  failure  of  the  alternative 
courses  of  action.  Such  a  total  risk  assessment  is 
beyond  the  charter  of  the  DDT&E  and  it  is  not  clear 
that  it  should  be  delegated  to  him.  Presently,  over¬ 
all  evaluation,  including  risk  assessment,  is  done  by 
all  the  DSARC  principals  by  means  of  the  lengthy  DCP 
coordination  and  prebrief  process.  While  this  process 
has  the  attribute  of  consensus  building  within  OSD,  it 
is  not  clear  that  it  is  the  most  efficient  way  to  gen¬ 
erate  an  independent  overall  evaluation  for  considera¬ 
tion  by  the  DSARC. 


It  is  also  observed  that  the  DSARC  spends  a  dis¬ 
proportionate  amount  of  time  and  energy  on  the 
milestone  III  review  at  which  point  they  have  rela¬ 
tively  little  opportunity  for  altering  either  the  sys¬ 
tem  concept  or  the  acquisition  strategy  in  the  face  of 
enormous  pressures  to  proceed  with  production.  Fur¬ 
thermore,  many  of  the  questions  regarding  the  adequacy 
of  T&E  data  arise  out  of  the  immaturity  of  the  hard¬ 
ware  subjected  to  the  DT&E  and  IOT&E  conducted  prior 
to  the  major  production  decision.  Present  policy 
states  that  milestone  II  includes  authorization  for 
limited  production  of  hardware  for  IOT&E.  However, 
there  appear  to  be  a  number  of  misconceptions  and  bar¬ 
riers  which  have  prevented  full  implementation  of  this 
approach.  If  this  policy  were  strictly  adhered  to,  it 
would  reduce  some  of  the  questions  regarding  adequacy 
of  T&E  prior  to  a  rate  production  decision.  It  would 
also  further  increase  the  importance  of  the 
milestone  II  review. 

( 5 )  Institutional  Pressures  Pose  Obstacles  to  The 

Developer  and  Tester  in  Planning  ano  Conducting 
an  Adequate  T&E  Program 

Institutional  pressures,  including  compe t ion  for 
resources,  cause  program  planning  to  be  success  ori¬ 
ented.  This  tends  to  minimize  the  flexibility  of  pro¬ 
gram  managers  to  deal  witf  R&D  problems  as  they  occur 
and  it  frequently  forces  compromises  in  the  adequacy 
of  planned  test  programs.  Furthermore,  resource  per¬ 
turbations  which  result  from  the  PPBS  process  are  not 
translated  into  changes  in  acquisition  strategies. 

PMs ,  trying  to  do  more  with  less,  invariably  run  out 
of  time  or  money  trying  to  generate  T&E  data  prior  to 
a  fixed  milestone  review. 

Current  practice  under  the  Uniform  Funding 
Policy,  which  allocates  costs  at  major  ranges  and  test 
facilities  to  individual  projects  rather  than  from  an 
institutional  fund,  aggrevates  this  problem  and  may 
inhibit  comprehensive,  thorough  testing  under  some 
c i rcumstances . 

2  .  RECOMMENDATIONS 

Exhibit  VI-1  shows  the  recommendations  of  this  study 
displayed  against  the  conclusions  just  discussed.  Each 
recommendation  is  described  below. 

( 1 )  The  DSARC  Should  Place  Greater  Emphasis  on 

Earlier  Milestones  in  the  Acquisition  Process, 
Particularly  on  Milestone  II 

It  is  clear  from  the  cases  studied  that  the  DSARC 
principals  were  placing  a  disproportionate  emphasis  on 
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MORE  SENSITIVE  TO  DOCUMENTING 
CHANGES  IN  EMPHASIS  AS  IT  OCCURS 


the  Milestone  III  re  view- -at  which  they  had  no  realis¬ 
tic  alternative  other  than  to  place  the  system  in  some 
form  of  production  irrespective  of  the  technical  or 
economic  risks.  The  emphasis  on  Milestone  III  is  cue 
partly  to  the  magnitude  of  resources  that  are  ap¬ 
parently  be  inq  committed  at  that  point;  to  the  inter¬ 
est  and  involvement  by  the  Congress  in  the  production 
decision;  and  to  the  tendency  of  the  decisionmakers  to 
focus  on  rather  near  term  decisions. 

From  a  practical  standpoint,  a  better  balance 
between  available  information  and  the  ability  of  the 
bureaucracy  to  influence  program  direction  occurs 
earlier  in  the  program  about  the  time  of 
Milestone  II.  This  is  where  the  major  commitment  to 
the  program  is,  in  fact,  made  and  this  is  where  sig¬ 
nificant  adjustments  to  the  acquisition  strategy  can 
be  made  without  major  economic  or  schedule  disrup¬ 
tions.  Furthermore,  this  is  about  where  the  critical 
transition  in  terms  of  the  magnitude  of  resource  ex¬ 
penditures  occurs.  While  a  decision  at  milestone  II 
necessarily  must  be  made  without  extensive  develop¬ 
mental  testing  (  and  hardly  any  meaningful  operational 
testing),  it  is  not  too  early  to  methodically  identify 
the  critical  test  issues  or  to  conduct  a  careful 
analysis  of  the  test  and  evaluation  plans. 

Shifting  the  DOD  emphasis  to  Milestone  II  will 
not  be  easy;  however,  policy  pronouncements  followed 
by  increased  vigor  and  intensity  of  Milestone  II  re¬ 
views  will  begin  the  necessary  acculturation. 

( 2 )  The  Policy  Regarding  the  Meaning  of  Milestones  II 

and  III  Should  be  Clarified 

Requiring  T&E  data  on  hardware  that  is  represen¬ 
tative  of  the  production  configuration  prior  to  the 
production  decision  is  a  classic  "Catch  22."  Present 
policies  address  this  problem  by  authorizing  initial 
production  for  the  purpose  of  initial  operational  test 
and  evaluation  at  milestone  II  and  full  scale  (or 
rate)  production  at  milestone  III.  However,  current 
practice  appears  to  segregate  the  R&D  and  production 
processes  to  the  point  that  multiple  milestone  III 
reviews  are  required,  i.e.,  a  five  milestone  system  is 
used  which  includes  a  separate  limited  production  as 
well  as  a  rate  production  decision. 

If  two  separate  production  reviews  are  desired, 
then  the  appropriate  directives  should  be  changed  to 
reflect  current  practice.  However,  if  authority  for 
initial  production  for  IOT&E  is  to  be  granted  at 
milestone  II,  then  the  present  policy  should  be  re¬ 
stated  and  emphasized.  Through  policy  issuances,  the 
Services  should  be  encouraged  to  develop  acquisition 

VI-  6 


V 


1 


s  tr atco ien  tr.at  ut  i  1 1  ear  ly  ;  r  ;t;;:cr  .  .  r.  naruvar-- 
IOT&K  ana  mat  requ  ire  ajtr.tntv  r  .  rata-  p  r  -a  jc  t 

-  n  l  3  *  x  :  1  ■  a:  t  (.>  r  :  i  ■  I  !  '  . 


New  k e s o -i  r  c Pr  e  r  oir.ir.  1  n<- 

.  h c  r. r.  i  vj u k-  ij  ji.b- *  -  r  e 

e  v  i  s  &  <  *  t  o  L  *-7  u  1  »\  i  ti  r.  *i  * .  ♦ 

Critical  T  r  u  r.  u  it  i  c  r.  •- r  m 

EDT&E  to  Production  Rune 
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In  t ho  cases  sturiiea ,  the  Services  -ana  ry  infer¬ 
ence  the  US  ARC)  frequently  felt  compelled  to  r.<*-lc  a 
Milestone  III  review  rather  than  risx  losira  prcctc- 
t ion  f unding  that  was  proy rammed  i  to  5  ye arc 
earlier.  The  fixed  milestone  date,  couplet:  witr.  gen¬ 
erally  .-ptimistic  planning  estimates  ar.c  normal  rc.1 
problems,  tended  to  cause  shortcuts  in  the  LT&E  ar.c 
IOT&E  needed  to  support  fully  a  recomrr.er.ciat  i  on  far 
rate  production.  Since  no  means  has  beer,  found  t-: 
eliminate  either  optimism  or  P.&D  problems ,  addec 
flexibility  is  required  tQ  adapt  tr.e  acquisition  pr  - 
cess  to  the  realities  of  the  PPBS  process. 

This  problem,  has  both  cultural  and  regulatory 
dimensions.  Differing  organizations  ar.c  procedures 
within,  the  DOD ,  and  to  some  extent  wit.-.m  tr.e 
Congress,  are  used  to  manage  ana  control  tr.e  EDI'iE  ar.c 
production  appropriations.  Tr.e  cr.asm  that  sometime:: 
exists  between  the  two  groups  and  tr.e  gene  rax  laco  of 
interchangeability  of  tr.e  two  types  of  funds  tends  to 
cause  program  managers  and  Service  proponents  to  ma<e 
inappropriate  compromises  in  Tit  programs  ar.c  tc  par¬ 
ticipate  in  budgeting  "gamesmanship"  ratter  than  deal 
with  the  realities  of  a  given  situation. 

If  Milestone  III  is  to  be  seriously  viewed  as  the 
decision  for  rate  production,  then  some  relief  must  re 
found  from  the  triangle  consisting  of  optimistic 
schedules,  inevitable  R&D  problems  and  milestone  re¬ 
view  dates  which  are  driven  by  a  desire  to  preserve 
follow-on  funding.  New  management  techniques  regard¬ 
ing  R&D  and  production  funding  would  allow  graceful 
slips  in  milestone  III  reviews  without  loss  of  funding 
until  a  legitimate  case  could  he  made  for  rate 
production . 

( 4 )  Greater  Care  and  Attention  Should  be  Giver,  to 


Defining  the 

Mean i ng 

of 

the  Term  " 

Operational 

Env  i  ronment " 

within 

the 

Context  of 

Eacr.  Major 

System  Acquisition. 

Throughout  this  study  of  DSARC  decis ionma< : rg , 
persistent  issue  arose  as  to  the  expectation  of  the 
decision  participants  over  the  requirement  to  test 
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systems  in  their  "Operational  Environment . "  In  none 
cases  technical  issues  were  involved  sue:,  as  numbers 
of  Air  Defense  Systems  or  the  complexity  of  the  elec¬ 
tromagnetic  environment  in  whicn  a  system  was  tested. 
In  other  instances  it  involved  the  use  of  contractor 
personnel  to  operate,  maintain  and  repair  the  system 
under  test.  Currently  employed  terminology  such,  a.: 

"as  near  as  possible  to  the  operational  environment" 
is  hardly  susceptible  to  universal  interpretation. 
Reaching  an  early  consensus  on  the  definition  of  the 
operational  environment  on  a  case- by -case  basis  would 
permit  better  T&E  planninq  and  eliminate  future  misun¬ 
derstanding.  (See  also  recommendation  *5  regardina 
disciplined  control  of  the  scope  of  the  system  to  re 
tested . ) 

( 5 1  More  Rigorous  System  Engineering  Discipline 


S  hou Id 

be 

Used  to  Establish  the  Scope  of  the  Sys 

tern  to 

be 

Tested  and  to 

Control  the  Impact  of 

Changes 

to 

Hardware  and 

Software  Configurations 

and  Test  Programs 


A  persistent  issue  between  the  services  and  the 
OSD  decisionmakers  is  the  scope  of  the  system  that  was 
tested,  i.e.,  how  much  of  the  entire  system,  including 
all  its  subsystems  and  related  or  associated  systems, 
was  or  should  be  tested?  In  what  degree  of  final  con¬ 
figuration  and  in  how  realistic  an  environment?  Down¬ 
stream  controversies,  particularly  at  milestone  III, 
could  be  minimized  by  means  of  system  block  diagrams, 
work  breaxdown  structures  or  other  system  engineering 
tools  to  depict  the  relationship  of  planned  tests  on 
the  system  in  its  entirety.  By  arraying  the  planned 
test  configuration  against  the  total  system  in  a  sys¬ 
tematic  way,  tradeoffs  between  cost,  schedule  and  risk 
could  be  realistically  assessed  and  future  ambiguities 
avoided.  On  larger  and  more  complex  programs  the  TEMP 
could  be  brought  under  formal  configuration  management 
control.  In  the  event  that  downstream  resource  allo¬ 
cations  required  readjustments  of  the  program,  inad¬ 
vertent  departures  from  established  expectations  could 
be  identified  for  coordination  with  DDT&E . 

( 6 )  The  TEMP  Should  Be  Viewed  as  the  Primary  Mec.n- 


anism  For  Structured 

Commun ica t i on 

Bet  wee  r,  OSP 

and  the  Services 

Rega  rd i na 

the  T&E 

Data  R  e  c  u  l  r  *?  d 

to  Progress  From 

One 

Phase 

of  Acaui 

s  i  t  i o r.  no 

Ano  the r 


The  TEMP  can  be  viewed  as  a  contract  ret vf---  .-V 

and  the  Services  in  which  the  adequacy  of  T&E  da-,  a  a* 
each  phase  of  the  process  is  defined.  However ,  i  *-  : 

important  to  recognize  that  the  contract  leni-  mat*-  ly 
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may  be  modified  by  either  party:  the  program  manager 
has  an  obligation  (consistent  with  his  repons ibility 
for  efficient  execution  of  the  program)  to  respond  to 
changing  situations  as  necessary,  and  OSD  must  focus 
on  making  good  decisions  in  the  light  of  existing  cir¬ 
cumstances  regardless  of  previous  commitments.  Thus, 
the  real  value  of  the  TEMP  is  as  a  means  of  imposing 
stuctured  communications  between  the  parties.  This 
means  that  the  TEMP  should  be  updated  whenever  changes 
occur  in  any  of  the  fundamental  underlying  assumptions 
on  which  the  T&E  program  is  based,  such  as  policy, 
funding  profile,  threat,  etc. 

( 7 )  Greater  Emphasis  Should  be  Placed  on  Identifying 
and  Analyzing  Alternative  Courses  of  Action  in 
the  Context  of  an  Overall  Evaluation 


A  singular  weakness  found  in  the  cases  studied 
was  the  lack  of  a  clear  cut  responsibility  for  bring¬ 
ing  toqether  the  many  factors  and  issues  which  require 
careful  balancing  for  informed  dec i s ionma k i na .  The 
present  process  depends  on  the  coordination  of  the  DCP 
to  reach  a  consensus--a  process  that  is  repetitive, 
time  consuming  and  sometimes  results  in  a  recommenda¬ 
tion  that  represents  the  least  common  denominator 
rather  than  the  optimum  course  of  action. 

There  are  alternatives  to  the  DCP  as  a  means  of 
conducting  an  overall  evaluation.  Responsibility  for 
this  function  could  be  assigned  to  an  existing  cr  new 
member  of  the  Defense  Acquisition  Executive's  staff, 
or  it  could  be  delegated  directly  to  the  program  mana¬ 
ger  who,  in  general,  has  greater  expertise  in  the  spe¬ 
cific  program  under  review.  In  any  case,  an  overall 
evaluation  would  include  technical,  affordability  and 
other  issues  of  which  the  independent  assessment  of 
the  DDT&E  is  but  one  part.  Consequently,  the  DDT&E 
assessment,  even  if  it  is  a  preliminary  or  partial 
one,  should  be  made  early  enough  to  be  seriously 
studied  in  an  integrated  review  of  all  factors. 

Finally,  an  overall  evaluation  should,  as  a  mat¬ 
ter  of  course,  identify  the  costs  and  other  ramifica¬ 
tions  of  each  of  the  alternative  courses  of  action, 
particularly  those  that  impose  delays  or  result  in 
uneconomic  production.  The  costs  of  such  decisions 
must  be  identified  and  consciously  acknowledged  as 
part  of  the  decisionmaking  process. 


( 8 )  Impact  Statements  Should  be  Required  From  Program 

Managers  of  Ma~;or  Systems  When  Changes  are  Made 
to  Defense  Acauisition  Policv 


One  finding  of  this  study  was  the  extent  to  w:.i~t. 
the  developmental  time  of  major  systems  ®xcee-beb  t:.e 
duration  of  the  acquisition  framework  under  wr.icr.  t :.ey 
were  carried  out.  Often,  policy  chances  wpn:  r.wt 
fully  reflected  in  programs  when  they  came  up  for 
milestone  review,  partially  because  resources  wt-r-  r.et 
provided  to  comply  with  the  policy  chance. 

Coordination  of  "for  comment  drafts"  is  one  way 
to  surface  such  problems  but  this  approach  apparently 
has  not  been  effective  in  avoiding  downstream  contrc- 
versv--either  because  the  questions  do  not  get  to  hr.c 
PMs  themselves,  or  because  their  answers  are  buffered 
by  the  Service  headquarters  staffs.  An  instition- 
alizea  process  whereby  PMs  prepare  specific  statement- 
on  the  impact  to  their  programs  of  changes  to  high 
level  acquisition  directives  would  deal  more  effec¬ 
tively  with  problems  of  retroactive  application  of 
policy  and  requirements  for  additional  resources  tr.ar. 
when  deferred  to  the  next  POM  or  ignored  altogetr.er. 
Such  statements  could  be  followed  by  or  made  part  of 
Program  Change  Requests  which  would  bring  the  effected 
programs  into  compliance. 

A  similar  but  more  difficult  problem  is  the  im¬ 
pact  of  changes  in  emphasis  by  defense  decisionmakers 
which  has  the  effect  of  policy  but  which  is  never  for¬ 
mally  communicated  to  the  Services.  In  these  cases, 
OSD  can  only  be  more  sensitive  to  documenting  def acto 
changes  in  policy  as  they  occur. 

( 9 )  The  DDT&E  Should  Make  Recommendations  Regarding 
Institutional  Improvements  in  T&E  as  Part  of  me 
DS ARC  Process 


Although  the  DT&E  has  important  responsibilities 
for  planning,  coordinating  and  budgeting  for  national 
ranaes  and  test  capability,  the  "lesson  learned"  from 
the  T&E  experiences  of  a  given  program  while  fresh 
•within  the  context  of  a  particular  DSARC  review  are 
not  immediately  fed  back  into  the  institional  manaae- 
ment  process.  The  independent  technical  evaluations 
provided  for  the  DSARC  are  important,  but  they  do  not 
help  in  managing  the  department  as  an  institution. 
Contemporaneous  recommendat ions  for  improving  the  T&E 
process  would  deal  more  effectively  with  the  broader 
institutional  issues  such  as  the  impact  of  T&E  funriina 
policies  on  system  acquisition  and  the  need  for  new  or 
improved  test  capab i 1 i t  ies  . 
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(10)  Continue  to  Monitor  the  Relationship  Between  T&E 
Dat3  and  DSAP  J  Decisions 


The  December  1979  revision  to  DODD  5000.3  and  the 
current  practices  of  the  DDT&E  encourage  a  more 
formal,  structured  dialog  between  OSD  and  the  Services 
regarding  the  adequacy  of  test  planning  and  data  as 
recommended  in  this  study.  However,  since  it  takes 
several  years  for  a  program  to  evolve  in  response  to 
policy  initiatives,  it  will  not  be  known  how  these 
policies  effect  the  frequency  of  conditional  decisions 
until  about  1982  or  1983.  Consequently,  the  issue  of 
adequacy  of  T&E  data  and  conditional  decisions  should 
be  monitored  to  determine  whether  structured  communi¬ 
cation  via  the  TEMP  is  effective. 

(11)  Determine  the  Underlying  Causes  of  Breaches  of 
Performance  Thresholds 


Most  of  the  conditional  decisions  that  occured 
during  the  period  under  study  could  be  attributed,  at 
least  in  part,  to  a  failure  to  meet  one  or  more  per¬ 
formance  thresholds.  An  investigation  of  the  underly¬ 
ing  causes  of  this  phenomena  was  beyond  the  scope  of 
this  study;  however,  there  is  a  large  potential  payoff 
in  gaining  an  understanding  of  how  and  why  performance 
breaches  contribute  to  increases  in  the  time  and  cost 
of  system  acquisitions. 

Failing  to  meet  a  perfomance  threshold  in  nearly 
90  percent  of  the  DSARCs  that  resulted  in  conditional 
decisions  indicates  more  systemic  problems  than  can  ce 
explained  simply  by  R&D  uncertainties.  Either  perfor¬ 
mance  specifications  are  being  set  unrealistically 
high,  i.e.  there  is  an  institutional  desire  to  achieve 
only  quantum  advances  in  technology,  or  milestone  re¬ 
views  are  being  held  too  early  in  the  acquisition 
cycle  to  allow  for  the  desired  degree  of  system  ma¬ 
turity,  or  the  DSARC  is  failing  to  appreciate  the  de¬ 
velopmental  nature  of  the  systems  under  review.  Tech¬ 
niques  such  as  performance  growth  curves  and  pre¬ 
planned  product  improvement  address  these  problems, 
however  a  detailed  study  of  the  underlying  causes  of 
performance  breaches  is  required. 
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Ma j .  Gen.  Powers,  Patrick  W. ,  "Streaml  in ma 
Army  Testing" 

Maj.  Gen.  Leaf,  Howard  W. ,  "Improving  AF 
Independent  Operational  Testing" 

Com.  Donovan  and  Lt .  Con.  Fitzgiobons,  "OT&E 
of  Ships:  Policy  and  Practice" 

Cpt .  Sherman  T.H.,  "T&E  in  Defense 
Procurement" 


Gleghorn  George  G. ,  "Optimizing  Test  Vehi 
Test  Programs" 

McCord  John  W. ,  "Management  of  Major  DOD 
Test  Facilities" 


Lt.  Col.  Washington  D.L., 
Equ ipment" 


T&E  of  TRI-TAC 


Kreiner  H.W. 


'How  Much  Testino  is  Encuah?" 


Ka rns,  Charles  W. ,  "Test  and  Evaluation 
Policy" 

Lt.  Col.  Butchho  M.  T.,  "Aircraft  OT&E" 

Lt.  Col.  Fritz,  Martell  D.,  "Use  of  Training 
Exercises  for  T&E" 


... —  t- 
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Love,  D.T.,  "Challenges  in  Aircraft  Engine 
Tj«E  Programs" 

Col.  Swett,  B.H.,  "Reliability  &  Maintenance 
for  the  Acquisition  Process" 

Rusterburg,  Owe- and  Geese,  "3-1  Structural 
Test  Program" 

Lt.  Col.  Mache  R.B.,  "Product  Liability." 

DSMC,  Schug  R.J.,  "DT&E  vs  OT&E  for  Naval  Air 
System  Command  Weapon  System  Acquisition,"  Study 
Project  Report  (DTIC  A043212,  May  1977) . 

SYNOPSIS:  The  purposes  of  this  study  are:  To 

review  current  directives,  reoorts,  factors,  and 
criticisms  that  determine  the  current  structure 
and  methodology  for  test  and  evaluation  within 
the  Navy;  to  determine  the  existence  of 
overtesting  or  inadequate  testing,  and  the 
emphasis  on  Operational  Test  and  Evaluation, 
which  affect  the  acquisition  of  weapon  systems  in 
the  Naval  Air  Systems  Command;  and  to  examine  t.oe 
extent  and  effectiveness  of  integrated  testing. 

QDDR&E/Def ense  Science  3oard,  "Report  of  Tack 
Force  on  T&E  Policies,"  February  17,  1977. 

SYNOPSIS:  An  earlier  Defense  Science  Board  Task 

Force  on  Test  and  Evaluation,  established  in 
1972,  had  summarized  and  delineated  general 
guidelines  for  Department  of  Defense  review  and 
monitoring  of  the  test  and  evaluation  aspects  of 
development  programs.  This  report  primarily 
considered  such  questions  as:  Is  testing  takino 
too  much  time?  Are  we  overtesting?  Is  all  of 
the  testing  worthwhile?  Are  we  estaol  ishi.no 
another  bureaucracy? 

The  Task  Force  concentrated  on  T&E  policies  by 
examining  a  small  group  of  representative 
programs  which  have  paid  .attention  to  test  and 
evaluation  from  the  office  of  the  DD  'T&E.  They 
concluded  that  T&E  is  a  fundamental  and  integral 
part  of  current  system  acquisition  procedures. 
Further,  they  found  little  or  no  overtest  ina 
being  done;  what  testing  was  done  contributed  to 
the  improvement  and  /er  i  f  ic.at  ion  of  svstem 
performance. 

DSMC,  Horik,  T.H.,  "The  Navy  T&E  Process  in  Mai 
Systems  Acquisition,"  Study  Project  Report  (DTI 
A035935),  November  1976. 


A- 4 


n  o 


SYNOPSIS:  Test  and  Evaluation  (T/E)  is  an 

integral  and  continuing  part  of  the  weapons 
system  acquisition  cycle.  In  the  process  of 
gaining  knowledge  from  research  through 
development  and  production,  some  T/E  occurs  each 
step  of  the  way.  In  the  past,  it  has  often  been 
compromised  as  attempts  were  made  to  meet  fixed 
deployment  dates.  This  study  investigates  the 
Navy's  T/E  process  in  the  procurement  of  ships 
and  aircraft,  including  some  recent  changes  in 
T/E  concepts  brought  about  by  recommendations  of 
the  Blue  Ribbon  Defense  Panel  and  DOD  Directive 
5000.1.  The  unique  problems  associated  with 
low-risk,  long-duration,  conventional  hull  ship 
acquisition  programs  are  discussed  including  Land 
Based  Testing,  Initial  Operational  Test  and 
Evaluation  (IOT/E),  and  DCP/DSARC  alternatives. 
Similar  problems  associated  with  high-risk  but 
relatively  short  aircraft  acquisition  programs 
are  also  addressed  with  emphasis  on  concurrency, 
prototyping  and  OT/E.  Incorporation  of  partial 
solutions  to  these  problems  {as  evidenced  in  the 
Patrol  Frigate  and  F-14  programs)  are  examined 
for  applicability  to  future  T/E  programs. 

ODDR&E/ Defense  Science  Board,  "Report  of  Task 
Force  on  T&E,"  April  2,  1974. 

SYNOPSIS:  The  Defense  Science  Board  Task  Force 

on  Test  and  Evaluation  was  established  at  the 
request  of  the  Director  of  Defense  Research  and 
Engineering,  on  behalf  of  the  Deputy  Director 
(Test  &  Evaluation)  to  develop  guidance  on  test 
and  evaluation  through  examination  of  a  group  of 
representative  weapon  systems  acquisition 
programs . 

The  resulting  report  presents  guidance  on  T&E  at 
two  distinct  levels.  At  the  most  general  level, 
the  report  discussed  a  number  of  issues  which  are 
appropriate  for  all  weapon  systems  acquisition 
programs,  and  are  generally  matters  of  basic 
policy.  These  issues  are:  reliability,  computer 
software,  human  factors,  the  "T&E  Gap", 
functional  specifications  versus  design 
specifications,  offense/defense  testing,  portable 
instrumentation,  ship  testing,  and  test 
planning.  In  addition,  the  report  includes  a 
general  checklist  of  items,  generally  applicable 
to  all  systems  development  and  deployment,  which 
is  organized  for  a  rapid  overall  review  of  T&E 
aspects. 
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.  DSMC ,  Brvder  W.D.,  "The  T&E  Evolution  (Relative 
to  USAF )  ",  "  Study  Project  Report,  DTI C  A040374  , 

May  1974. 

SYNOPSIS:  The  Blue  Ribbon  Defense  Panel  Report 

of  July  1970  provided  the  impetus  for  major 
changes  in  the  structure  and  conduct  of  test  and 
evaluation  (T&E)  in  the  Department  of  Defense. 
This  report  traces  the  actions  taken  by  OSD,  Air 
Force  and  Systems  Command  over  the  last  four 
years  up  to  the  formation  and  implementation  of 
the  Air  Force  Test  and  Evaluation  Center  (AFTEC) 
in  January  of  1974.  To  ascertain  the  probable 
direction  of  T&E,  several  members  of  the  T&E 
community  were  interviewed.  The  results  of  these 
interviews  are  presented  and  discussed.  They 
indicate  a  uniform  and  stable  policy  throughout 
the  Air  Force  and  an  increasing  role  of  AFTEC  in 
the  T&E  business  as  they  become  active  and 
involved  in  the  emerging  programs.  No  major 
changes  are  anticipated  in  the  current  T&E 
policies  and  directives. 

.  GAO  Report,  "The  Importance  of  T&E  in  the 

Acquisition  Process  for  Major  Weapon  Systems," 
August  1972. 

SYNOPSIS:  This  report  examined  T&E  on  13  weapon 

systems.  It  concluded  that  most  systems  did  not 
have  adequate  test  plans,  T&E  was  not 
accomplished  in  a  timely  manner,  T&E  data  was  not 
available  prior  to  those  times  in  the  acquistion 
cycle  when  decisions  had  to  be  made.  The  report 
also  addressed  waiver  problems.  The  GAO 
recommended  completion  of  T&E  prior  to  key 
decisions,  adequate  controls  over  T&E  waivers  and 
improved  reporting  of  T&E  results.  Some  believe 
the  report  contributed  to  the  "T&E  explosion." 

DEVELOPMENTAL  T&E 

.  GAO  Report  PSAD-79-86,  "Effectiveness  of  DCD 
DT&E , "  June  25,  1979. 

SYNOPSIS:  In  1978,  GAO  reviewed  the  military 

services'  development  test  and  evaluation  (DT&E) 
of  six  weapons  systems  totaling  an  estimated  $12 
billion  in  development  and  procurement  costs. 
Inadequate  testing  is  principally  due  to  the 
expense ,  the  difficulty  of  staging  and  execution, 
the  time  consumed,  and  belief  that  negative  test 
results  may  jeopardize  or  delay  a  program.  To 
overcome  the  unenthus  iastic  attitude  toward 
testing,  the  type  of  results  expected  from  the 
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testing  must  be  defined.  Possibly,  a  strong 
testing  activity  is  the  only  means  of 
accompl ishing  this  objective.  The  Department  of 
Defense  spelled  out  testing  procedures  in  a  197 
do  1 icy . 

This  study  found  that,  as  design  and  development 
phases  progressed,  the  programs  experienced  cost 
growth,  schedule  slippages,  and  a  reduction  in 
performance  levels.  To  remedy  these  effects,  fr.« 
responsible  agencies,  without  formal  assessment 
of  risks  and  benefits,  reduced  the  scope  of,  and 
hardware  for,  DT&E  during  critical  tests; 
advanced  programs  into  operational  testing  and 
production  before  completion  of  DT&E;  and 
approved  concurrent  development  and  production; 
all  without  formal  assessment  of  risxs  and 
benefits.  In  five  of  the  six  programs,  planned 
tests  for  resolving  technical  uncerta  int  ies 
either  were  reduced  or  not  performed,  and  final 
test  reports  either  were  unavailable  or  omitted 
the  critical  issues  to  be  resolved  before  a  given 
development  phase  could  be  considered  complete. 
These  practices  could  result  in  the  systems 
entering  production  before  their  performance 
capabilities  are  known.  Judging  from  past 
experience,  this  is  an  unsat isf actorv  procedure 
because  systems  have  emerged  from  testing  with 
degraded  performance  or  with  additional  costs 
incurred  in  raising  performance  levels  to 
acceptable  standards.  Actions  which  would  reduce 
development  time  and  costs  include  the 
elimination  or  reduction  of  development  tests  of 
systems  and  subsystems,  the  combination  of  D'T&E's 
with  initial  operational  test  and  evaluation 
( IOT&E) ,  and  a  clear  assessment  of  the  risks 
involved  in  concurrent  development  and  production. 

The  recommendations  of  this  report  to  the 
Agencies  are  that  the  Secretary  of  Defense  should 
require  that  DT&E  reports  identify  any  deleted, 
reduced,  or  substituted  tests,  and  state  clearly 
the  risks  and  implications  of  these  actions.  The 
Secretary  should  also  require  test  plans  to  show 
the  additional  risks  of  joint  DT&E/'IOT&E  test 
schedules  and  allow  time  to  correct  discovered 
deficiencies  before  proceeding  to  the  next 
testing  phase.  Formal  assessment  of  risks  and 
benefits  should  be  required  before  approval  of 
concurrent  development  and  production,  and 
developers  should  be  ordered  to  monitor  sensitive 
system  elements  3nd  high-risk  areas  closely,  and 
provide  schedule  and  funding  reserves  to  resolve 
problems  as  they  are  identified. 
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OPERATIONAL  T&E 


GAO  Report  PSAD-.30-2,  "Army  Op* rat  ionai  Test 
Evaluation  Needs  Improvement s"  (CONFIDENTIAL 
Novemoo r  13,  14  70 . 

GAO  Report  PSAD-79-1,  "Fo  I  Low-  on  OT&E  of  W  e  a 
Systems,"  October  19,  1979. 


SYNOPSIS:  A  surve-. 


or  or  act ic 


in 


for  follow-on  operational  testing 
of  weapon  systems  by  the  military  s 
indicated  that  follow-on  tests  have 


and  evaluat 


:~j  &  &  p 


conducted  aueoustel'. 


However,  c; 


where  critical  testing  was  deferred  until 
production  decision  was  made,  and  fail-seal 
operational  test inc  has  been  conducted  af  he 
weapon  systems  have  been  produced  and  ieola 
In  the  Army,  critical  tests  needed  to  suddo 
low- rate  production  decision  were  deferred 
after  production  units  were  available.  Ir. 
Air  Force  most  operational  testina  trior  to 
initial  production  decision  is  combined  wit 
testina  performed  oy  developers;  this  does 
constitute  sufficient  operational  testing. 
Unlike  the  Army  and  the  Air  Force,  the  Navy 
usually  performs  sufficient  operational  tes 
before  production  decisions,  but  the  "  zy  a 
continues  to  conduct  full-scale  operatior.al 
testir,g  after  systems  have  been  ieploved. 
ooe rational  testina  of  comolex  veaoon  svste 


arc  o : 


w  i  1 


;oe  necessarv  after  orodi 


.ion  iec : s  i  o n  * 


when  t.nere  are 

r  i 


:nanc< 


o  r 

f 


made;  this  could  occur 
threat,  major  system  mod  it icat ions , 
doctrine  or  tactics.  The  Secretary  or  uer 
should:  assure  that  ope  r  a  t  i  ona  tests  c-  it 

to  determining  system  ef f ect i and 
suitability  are  .accomplished  prior  to  init: 
production  decisions;  and,  in  cases  where  = 
have  undergone  full  operational  test  and 
evaluation,  restrict  additional  testing  to  ; 
of  significant  changes  in  mission,  threat, 
tactics,  or  system  modifications. 

GAO  Reoort  PSAD  "3-131,  "Goer. at :  anal  Lett  a: 
Evaluation  of  Fore i an  Built  Weapon  S. -stems, 
Julv  25,  1973. 


GAO  Report  PSAD-73-"7,  "Navy  OTsE:  A  7 
Tool  Not  Fully  Utilized"  < 20MF I DENT I A L ) 
March  29,  1973. 
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SYNOPSIS:  The  Navy's  Operational  Test  am 

Evaluation  Force  tests  anti  eval  rates  N  avy  ipor. 
systems  in  an  operational  environment  am 
develops  procedures  and  tactics  tor  the :  r  it'-. 
The  Test  Force  is  separate  and  independent  of 
developing,  procuring,  and  using  commands,  an: 
reports  results  of  operational  tests  and 
evaluations  directly  to  the  Chief  of  Naval 
Oper at  ions . 

The  Navy  nas  made  considerable  progress  ir. 
planning  and  conducting  tests  and  report:-.:  m-'t 
results.  However,  in  some  instances,  Nav 
official,  made  nrocurement  decisions  he  for 
sufficient  operational  test  and  evaluation  w a s 
completed,  ana  some  systems  entered  production 
be?  ore  it  was  known  whether  they  could  perform 
their  missions  under  combat  conditions. 
Operational  test  and  evaluation  results  complete- 
on  some  weapon  systems  after  production  began 
showed  significant  performance  problems. 
Deviations  from  prescribed  testing  procedures 
should  be  made  only  when  the  need  for  the  syst-m 
is  critical  or  the  risk  of  performance  problem? 
is  minimal.  In  some  cases,  a  complete 
operational  test  and  evaluation  could  not  be  mad*: 
because  of  limited  test  resources. 

Recommendations  of  tr.is  report  to  Conaress  are 
that  the  Secretary  of  Defense  should  demonstrate 
through  operational  testing  that  weapon  systems 
can  perform  their  missions  in  typical  combat 
environments  before  they  are  approved  for 
production  and  see  that  the  Navy's  Operational 
Test  and  Evaluation  Force  is  given  adequate 
resources  to  perform  the  necessary  test  and 
evaluation. 

T&E  GUIDELINES 

.  As  an  outgrowth  of  the  Defense  Science  Board  Tan- 
Force  on  Test  and  Evaluation,  the  DD/T&E  in  April 
197  4  com.oiiji  •  series  of  "T&E  Guidelines"  tor 


various  categories  of  weapon  systems, 
included : 


T&r  Gul  i'-lm-'  for  Missile  Weapon  S' 


T&E  Guid»l ;  p.»3  for  Aircraft  Xeaoo: 


>  ''stems 


.  &  r.  G  u  :  :  o 


:  •  r  mnio  bvs terns 


T&n  Gu  i  do  1  i  s  for  Ground  Venicle  S  vs  tor 


TSE  Guidelines  for  ASW  System 3 

r&E  Guidelines  for  Airborne  ECM  Sy::t--. 

T&E  Guidelines  for  Airborne  Genera] 
Surveillance  Radar  Systes 

T&E  Guidelines  for  Command  and  Centro 
Systems 

T&E  Guidelines  for  Common  Test  G  e  a  . 

OPERATIONAL  SUITABILITY 

OUSDRE  -  DDT&E/0ASDMRA&L/W5,  "Operational 
Suitanility  Verification  Study,  Phase  I  Fi 
Report,"  January  1930. 

SYNOPSIS:  This  report  presents  the  result 

study  of  the  weapon  system  T&F  oroc«ss  r-1 
to  operational  suitability.  The  study  iev 
policies  and  procedures  which  a i  i  in  ear  lv 
formulation  of  operational  su.ta.oility 
requirements.  Included  were  olanning  a: 
comprehensive  and  effective  test  oroorams, 
exploit  the  time  and  resources  mace  avails 
testing,  and  accurate  assessment  of  system 
operational  suitability  cnaracter  1st  its. 

.  GAO  Report  PSAD-91 - 17 ,  "Effectiveness  of  " 
Forces  can  be  Increased  Thr  ough  Impr  0 :  v; 
System  Design,"  January  29,  1930. 

SYNOPSIS:  This  report  discusses  weusno.cso 

the  Department  of  Defense's  major  system 
acquisition  process  which  are  con t r i ou t i no 


ope  ration, 

ma 

intenance,  and  support 

pronl  e.m 

many  deplov 

zed 

weapon  systems.  To  is 

rev : tw 

undertaken 

in 

an  effort  to  identify 

c  h  a  n  -7  ^  s 

are  needed 

in 

the  acquisition  orece 

c  ;r  **  O  Ti  i 

problems  with 

future  weapon  systems 

• 

MAJOR  SYSTEM  ACQUISITION 

GAO  Report  PSAD-79-106,  "Rev iev;  of  the  DOD 
Implementat ion  of  Procurement  Reforms," 
September  25,  1979. 

GAO  Report  PSAD-79-80 ,  "Recommendat  i-'rs  •>:] 
lommission  on  Government  Procurement :  A  F 
Assessment,"  May  31,  1979. 


SYNOPSIS:  As  Federal  nrocu  rement  expanded,  and 

individual  problems  multiplied,  a  patchwork  of 
laws,  directives,  and  reaulat ions  was  installed 
to  cope  with  the  problems.  The  Congress,  after 
exhaustive  hearings  in  the  late  1900's,  create: 
the  Commission  on  Government  Procurement  to 
devise  fundamental  improvements.  This  report  is 
the  eighth  status  report  on  the  recommendations 
proposed  by  the  Commission  on  Government 
Procurement  in  December  1972. 

GAO  Report  PSAD-79-9,  "Observations  on  OMB 
Circular  A-109,  Major  System  Acquisition  by  DOD ,  " 
February  20,  1979. 

OUSDR&S/De Cense  Science  Board,  "Report  of  the 
Acquisition  Cycle  Task  Force,"  March  15,  1978. 

SYNOPSIS:  This  Defense  Science  Board  Summer 

Study  TasK  Force  on  the  Acquisition  Cycle 
examined  a  numoer  of  major  systems  acquisition 
case  histories,  changes  in  acquisition  policy 
over  the  past  two  decades,  and  the  actual 
workings  of  the  program  advocacy  and  budgetarv 
processes.  The  Task  Force  concluded  that,  over 
the  past  15-20  years,  the  acquisition 
process--f rom  initial  program  concept  to  initial 
operational  capabilitv--has  increased  in  length 
by  approximately  five  years  on  the  average.  Most 
of  this  increase  has  been  in  the  "front  end"  of 
the  process,  from  the  time  of  initial  concept 
studies  to  the  start  of  Full  Scale  Development 
(i.e.,  DSARC  II)  and  at  the  "back  end"  from  the 
completion  of  FSD  at  DSARC  III  to  the  achievement 
of  a  full  operational  capability  (i.e.,  the 
production  and  deployment  phase). 

The  Task  Force  also  concluded  tnat  full-scale 
prototyping  often  introduces  costly  and 
unnecessary  program  delays;  that  the  policy  of 
"no  concurrency"  is  being  applied  too  rigidly  and 
is  inefficient  and  costly  in  many  cases;  and  that 
sequential  and  separate  testing  systems  by 
developers  and  by  the  users  (instead  of  the 
independent  evaluation  of  jointly  obtained  test 
data)  often  adds  unnecessary  expense  and  time. 

It  was  also  concluded  that  the  practice  of 
incremental  decisionmaking  by  reauirina  the 
achievement  of  a  consensus  between  the  advocates 
and  the  antagonists  at  each  DSARC  milestone  is  3 
cumbersome  process  that  tends  to  prolong  programs 
so  much  that--during  the  acquisition  cvcle--the 
threat,  the  players ,  and  even  the  conceDt s  of 
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warfare  often  change,  resulting  in  the  necessity 
to  cancel  programs  and  start  new  ones  to  meet  the 
threat,  with  the  concomitant  waste  of  limited 
national  resources. 

Finally,  the  Task  Force  expressed  concern  that 
without  very  careful  attention  to  the 
implementation  of  OMB  Circular  A- 109,  this 
well-intentioned  policy  could  very  likely  give 
rise  to  institutionalized  procedures  that  w .  ! 
significantly  increase  the  length  of  the  "front 
end"  phase  of  program  acquisition  rithet  that 
shortening  it  as  intended  and  without  achieving 
the  intended  degree  of  program  flexibility  and 
stability  in  acquisition  policy. 

.  GAO  Report  PSAD-73-14,  "A  Critique  of  toe 

Performance  of  the  Defense  Systems  Acquisition 
Review  Council,"  January  30,  1978. 

G.  CONGRESS 

.  GAO  Report  PSAD-80-37,  "SARs:  Defense  Department 
Reports  That  Should  Provide  More  Information  to 
Congress,"  May  9,  1980. 

SYNOPSIS:  This  report  points  out  ways  in  which 

the  Department  of  Defense's  selected  acquisition 
reporting  system  could  be  changed  to  present  more 
completely  the  status  and  progress  of  major 
weapon  systems  and  their  expected  operational 
capabilities  and  limitations.  This  review  was 
made  at  the  request  of  the  Chairman,  Subcommittee 
on  Priorities  and  Economy  in  Government,  Jo.nt 
Economic  Committee. 


GAO  Report  PSAD-79-46 ,  "Need  for  More  Accurate 
Weapon  System  Test  Results  to  be  Reported  to  t r •’ 
Congress,"  March  9 ,  1479. 

SYNOPSIS:  The  results  of  the  tests  and 

evaluations  of  10  Navy  and  5  Air  Force  w-  vo.-,-. 
systems  contained  in  the  Data  Sheets  reviewed  rv 
GAO  v;ere  found  no  be  incomplete,  m  i l-'ad  i  no  , 
and/or  outdated.  The  three  Army  systems  select*-'  i 
for  review  had  deficiencies  identified  •urmo 
recent  operational  testing ;  however,  thus*3  test 
results  were  satisfactorily  reported:  t  h-  Data 

Sheets.  The  following  factors  have  contributed 
to  inaccurate  reporting  of  test  results  to 
Congress:  reviewers  at  higher  levels  often  omit 

or  change  t  ie  Congress i ana  1  Data  Sne»t  ~  prepare  i 
by  independent  test  agencies;  the  reports  are  not 


updated  between  the  annual  reporting  periods;  2 
the  Department  of  Defense  has  not  provided 
.sufficient  instructions  regarding  the  type  of 
information  required  in  Congressional  Data  Shoe 

Recommendations  include  that  the  Secretary  of 
Defense  should  improve  the  quality  and  timeline 
of  operational  test  results  reported  in 
Congressional  Data  Sheets  through  the  followino 
procedures:  (1)  mor--*  thorough  review  of 

Congressional  Data  Sheets  by  the  Office  of  the 
Director  of  Test  and  Evaluation  before  submiss: 
to  Congress;  (2)  more  definitive  guidelines 
concerning  the  kind  of  test  information  that  is 
required  in  Congressional  Data  Sheets;  and  (3) 
continual  evaluation  and  identification  of  new 
results  that  become  available  after  initial 
Congressional  Data  Sheets  have  been  submitted  1 
order  to  determine  if  updated  information  shoal 
be  reported  to  Congress. 

MISCELLANEOUS 

.  GAO  Report  PSAD-30-61,  "Implications  of  Highly 
Sophist icated  Weapon  Systems  on  Military 
Capabilities,"  June  30,  1930. 

SYNOPSIS:  This  report  identifies  the  hionlv 

sophisticated  nature  of  deployed  weapons  as  a 
contributor  to  budgetary  problems,  inventory 
shortfalls,  and  a  low  state  of  readiness  'or 
certain  combat  categories.  It  points  out  that 
Defense  efforts  to  introduce  lower  cost  weapon 
systems  that  are  more  reliable,  available,  and 
maintainable  have  not  been  as  successful  as 
desired.  The  -eview,  requested  by  the  chairman 
of  the  Senate  Committee  on  Foreign  Relations,  w 
prompted  by  the  November  8,  1979,  report  on 
"Impediments  to  Reducing  the  Costs  of  Weapon 
Systems,"  PSAD-60-6. 

.  GAO  Report  PSAD-80-43,  "Issues  Identified  in  21 
Recently  Published  Major  Weapon  System  Reoorts, 
June  12*,  1930. 

SYNOPSIS:  This  report  summarizes  21  major  we an 

system  reports  issued  during  January  and  Februa 
1980  .  The  purpose  is  to  focus  attention  on  t 
principal  issues  that  were  found  to  be  common 
among  several  weapon  programs.  The  report  a  s  ' 
serves  as  a  quick  reference  to  all  major 
acquisition  work  during  the  past  12  months. 
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GAO  Report  PSAD-80-6,  "Impediments  to  Reducing 
the  Costs  of  Weapon  Systems,"  November  3,  1979. 

SNYOPSIS:  GAO  believes  the  major  effects  on  the 

costs  of  weapons  systems  have  resulted  from:  1 1 ■ 

attempts  to  deploy  systems  with  new  technoloav 
and  high  performance;  (2)  low  rates  of  production 
due  to  budget  constraints  and  desires  to  maintain 
active  production  bases  as  long  as  possible;  '3 
absence  of  price  competition  between  contractors; 
(4)  lack  of  real  motivation  on  the  part  of 
contractors  to  reduce  costs;  (5)  the  impact  o f 
socioeconomic  programs,  government  controls,  an.'5, 
red  tape;  and  (6)  a  nationwide  problem  of  reduced 
research  and  development  expenditures  and 
lessening  productivity. 

Recommendations  made  in  this  report  include  that 
the  Secretary  of  Defense  should  make  a 
comprehensive  study  to  identify  those  aspects  of 
contract  administration  that  can  be  relaxed  or 
modified  in  order  to  reduce  costs  and  paperwork. 
The  Secretary  should  also  take  stronger 
initiatives  to  accelerate  the  implementation  of 
management  policies  for  major  weapon  system 
acquisitions,  as  set  forth  in  the  Office  of 
Management  and  Budget  (OMB)  Circular  A-I09.  The 
Armed  Services  and  Appropriations  Committees 
should  carefully  examine  lower  cost  options 
before  approving  new  weapon  programs.  In 
particular,  the  committees  should  explore  with 
senior  military  officials  the  pros  and  cons  of 
larger  quantities  of  alternative  weapons  versus 
sma 1 le r  numbe rs  of  highly  sophisticated  and 
expensive  systems.  The  Committees  also  should, 
after  being  satisfied  that  a  weaoon  system  is 
ready  for  production,  consider  multiyear  fundina 
in  order  to  take  advantage  of  more  economical 
production  practices.  The  Congress  should  take 
the  initiative  in  responding  to  the 
recommendations  of  the  Commission  on  Government 
Procurement  to:  (1)  reexamine  the  full  range  of 

socioeconomic  programs  applied  to  the  procurement 
process  and  the  administrative  practices  followed 
in  their  application;  and  (2)  raise  the  minimum 
dollar  thresholds  at  which  such  programs  are 
applied  to  the  procurement  process. 

GAO  Report  PSAD-79-64,  "Digests  of  Major  Weapon 
System  Reports,  Issued  January  and  Februarv 
1979,"  April  25,  1979. 
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SYNOPSIS:  This  report  contains  unclassified 

digests  of  major  weapon  system  reports  issued 
during  January  and  February  1979  and  a  listing  of 
major  acquisition  reports  issued  from  July  1973 
through  March  1979.  Selected  reports  are  included 
in  the  appropriate  bibliography  section. 

III.  WEAPON  SYSTEMS  REPORTS 

A.  AEGIS 

GAO  Reports 

.  GAO  Report  PSAD  80-13,  "Status  of  tno  Navy's 
Aegis  Weapon  System  and  DDG-47  Shipbuilding 
Program,"  February  28,  1930. 

.  GAO  Report  PSAD-78-29,  "Status  of  the  Aeais 
Program"  (SECRET) ,  February  24,  1973. 

Aviation  Week  and  Space  Technoloav 


’Senate  Agrees  to  Cancel  Nimitz,  Equip  Long  Beacr 
With  A'-gis,"  March  14  ,  1977,  o.  22. 

.  Filter  Center:  Shipboard  combat  system  that 

integrates  Aegis  with  ether  weapon  systems  to  no 
developed  by  RCA,  November,  1476,  p.  45. 

.  Industry  Observer:  Navy  will  equip  surface  sk.rs 
with  Aegis  air  defense  system.  May  3,  1976,  o.  11. 

Other  Periodicals 

.  Prina,  L.  Edgar,  "Defender  of  the  Fleet  and  of 
the  Not-So-Fl eet , "  Sea  Power,  Vol.  23,  April, 

1980 ,  pp.  60-62. 

.  _ ,  "Shipboard  Voice  Communications  Seine 

Upgraded  With  Integrated  System,"  Defense 
Electronics,  Vol.  11,  July,  1979,  po.  46-45. 

.  Polman,  Norman,  "USS  Norton  Sound,  the  Newest  Old 
Ship,”  US  Naval  Institute  Proceedings,  Vol.  105, 
April,  1979,  pp.  79-83. 

.  Cdr.  Blades,  Todd,  "DDG-47,  Aegis  on  Its  Wa-  to 
Sea, "  US  Naval  Institute  Proceedings ,  Vo  1 .  105, 

January,  1979,  pp.  101-106. 

.  DeVore,  Charles,  "Acuic:  Shield  of  the  Fl-=^t  of 
the  1980's,"  S  i  c n  a  1 ,  Vol.  33,  October,  1 0  -  3 , 
pp.  46-47. 
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Hartman,  Richard,  "Aeq is,  Navy's  New  AA'W  Weapo 
System,"  Military  Electronics  '  Coer,  te  rme  a  c  j  res 
Vo  1 .  3,  September,  1^77,  o.  14. 

Meyer,  Wayne  E.,  "Aegis,  RAD.M , "  VS  Naval 
Institute  Proceedings,  Vo  1.  10  3,  Feoruary,  l07 
po.  93-97. 


Taylor,  T.  W.  and  Wares,  E.  F. ,  "A  Sophist ic  at 
Test  System  to  Support  Modern  Weaponry,"  Def  en 
Management  Journal,  Vol.  13,  January,  1Q77,  op 
16-23. 


B .  ALCM 


GAO  Reports 

.  GAO  Report  PSAD-79-19,  "Air-,  Sea-,  and 

Ground- launched  Cruise  Missile  Program  Status 
Issues"  (SECRET  MOFORN, ,  February  12,  1979. 

GAO  Reoort  PSAD-78-39,  "Status  of  the  Cruise 
Missile  Program"  (SECRET),  April  23,  1°73. 

.  GAO  Report  PSAD-77-36,  "Status  of  Air  and  Sea 
Launched  Cruise  Missile  Programs"  (CONFTDENTIA 
April  26,  1977. 

Aviation  Wee<  and  Soac^  Technology 

.  "ALCM  Crash  Delays  Program,"  December  3,  la30, 

p.  22 . 

"ALCM  Test  Fails,"  July  28,  1930,  p.  1". 

.  "Boeing  ALCM  Tested,"  June  23,  1980,  p.  13. 

.  "ALCM  to  Enter  Inver  'v , "  June  16,  1930,  p.  7 

.  "Boeing  Gears  for  Production,"  May  12,  1930, 
p.  63 . 

.  "Software  Key  to  ALCM  Choice,"  March  31,  1930, 

p.  18 . 

.  "ALCM  Contractor  to  Be  Selected,"  February  13, 

1980,  p.  22. 

.  "'Weather  Could  Prevent  Flyoff,"  December  3,  1° 

p.  22 . 

.  "Cruise  Missiles  in  Flight  Tests,"  October  l7, 
1979,  p.  74. 
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"ALCM  Test  Terminated,"  October  15,  1979,  d.  22 

"Boeing  Probes  Crash,"  August  13,  1979,  p.  18. 

""ruise  Missile  Flyoff  Begins,"  July  30,  1979, 
p.  14. 

"Disperser  Modified  for  ALCM,"  April  2,  1979, 

p .  6  6. 

"First  ALCM-B  Test  Missile  in  Fabrication," 

July  24,  1978,  p.  61. 

"Cruise  Missile  Launch  Concepts  Detailed," 

June  12,  1978,  p.  56. 

"USAF  Plans  Radar  Guidance,"  May  15,  1973,  ?.  1 

"ALCM  Undergoing  Tests,"  February  20,  1973,  p. 

"Tomahawk,  ALCM  Survivability  Tests," 

December  12,  1977,  p.  17. 

/ 

"Mew  Air-Launched  Cru’Se  Missile  Mode  Shown," 
December  5,  1977,  o.  17. 

"Cruise  Missile  Flyoff  Planned,"  July  18,  1977, 

p.  12 . 

"Competitive  Cruise  Missile  work  Urged,"  July  1 
1977,  p.  22. 

"Boeing  Set  for  Cruise  Missile  Speedup,"  July  i 
1977,  p.  19. 

"USAF  Pushes  Long-Range  Cruise  Missile  Version, 
April  18,  1977,  p.  19. 

"Boeing  USAF  Missile  Effort  Set,"  March  14,  19” 
p.  23. 

"Cruise  Missile  Order,"  January  24,  1977,  p.  17 

"USAF  May  Accelerate  Missile  Programs," 

October  ll,  1976,  p.  14. 

"ALCM  in  Flight  Tests,"  July  12,  1976,  o.  19. 

"WRU:  DOD  to  Study  2  Strategic  Weapons  Plans," 

May  31,  1976,  p.  11. 

"ALCM  Maneuverability  Shows,"  May  24,  1976,  p. 
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)ther  Periodicals 


M*»yer ,  Deoor ah  G.  ,  "Boeinq  Lands  ALCM  Contract," 
Armed  Forces  Journal  International ,  Vo  1 .  117, 

May  1980,  p.  17. 

Mag.  Robinson,  Gerald  L. ,  "ALCM  Construction 
Program,"  Air  Force  Engineering  and  Services 
Quarterly ,  Vol.  21,  May  1980,  op.  8-30. 

,  "New  Missile  (ALCM)  Picked,"  Air  Force 
Times,  Vol.  40,  April  7,  1930,  p.  7. 

Meyer,  Deborah  G. ,  "ALCM  Competitors  Complete 
Flight  Tests,"  Armed  Forces  Journal 
International,  Vol.  117,  March  1980,  p.  31. 

Lopez,  Roman,  "ALCM- -Who  Will  Build  the  ALCM," 
Interavia,  Vol.  35,  March  1980,  pp.  206-208. 

_ "Cruise  Missile  Development:  The 

Competition  Is  Fierce  Everywhere,"  Gove r nmen t 
Execut ives ,  Vol.  12,  January  1980,  po.  22-24. 

Maj.  Tucker,  Charles  G.,  "The  ALCM  Decathlon," 

Air  Force  Magazine,  Vol.  62,  June  1979,  pp.  67-71. 

Berry,  Jr.,  F.  Clifton,  "Cruise  Missile 
Developments  in  the  Aftermath  of  SALT  II," 
Interavia ,  Vol.  34,  August  1979,  pp.  760-764. 

Lamb,  J.E.,  "Cruise  Missile  -  Superweapon," 
National  Defense,  Vol.  62,  SeDt-Oct  1977, 
pp.  120-122. 


Brown,  Harold,  "The  3-1  Cruise  Missile  Decision," 
Commanders  Digest,  Vol.  20,  September  15,  1977, 
Entire  Issue. 


Clements,  Williams  P. ,  "View  From  the  Top  (Depot- 
Secretary  of  Defense),"  Countermeasures,  Vol.  3, 
Jan  1977,  p.  6. 


Berry,  F.  Clifton,  "ALCM  Flying  Hiqh--Is  It 
Doomed?"  Armed  Forces  Journal  International, 
Vol.  114,  Oct  1976,  p.  22 


C.  EF-111A 


GAO  Reports 


GAO  Report  PSAD-30-16,  "Lack  of  Full  Disclosure 
of  EF-111A  Program  Cost,"  December  4,  1979. 
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.  GAO  Report  PSAD-80-71,  "Evaluation  of  EF-111A 
Extended  Development  and  Full-Scale  Production 
Decision,"  August  27,  1980. 

.  GAO  Report  PSAD-79-5,  "Assessment  of  Joint 
DT&E/10T&E  Results  of  EF-111A"  (SECRET) , 

November  3,  1978. 

.  GAO  Report  PSAD-79-74,  "Recommendations  for 

Additional  Testing  Prior  to  Full-Scale  Production 
of  EF-111A,"  April  25,  1979. 

.  GAO  Report  PSAD-78-130,  "Recommendations  for 
Improving  Testing  of  EF-111A  Tactical  Jamming 
System,"  June  30,  1978. 

.  GAO  Report  PSAD  78-32,  "Status  of  the  Air  Force's 
EF-111A  Tactical  Jamming  System"  (SECRET), 

March  23,  1078. 

.  GAO  Report  PSAD-77-25,  "Status  of  the  EF-111A 
Electronic  Warfare  Aircraft  Program" 
(CONFIDENTIAL),  February  3,  1977. 

Aviation  Week  and  Space  Technology 

.  "EF-111A  Conversion  to  Start,"  August  11,  1980, 

p.  25. 

"EF-111A  Exceeds  Goals,"  October  29,  1979,  p.  68. 

.  "EF-111A  Costs  Cause  Concerns,"  Julv  17,  1973, 

p.  15. 

" USAF  Improves  ALQ-99,"  February  6,  1978,  p.  97. 

.  "EF-111A  Geared  for  Electronic  Thicket," 

February  6,  1978,  p.  93. 

"EF-111  Electronic  Warfare  Aircraft,"  Februarv  6, 
1978,  p.  56. 

.  "ECM-11  in  Tests,"  March  28,  1977,  p.  58. 

Other  Periodicals 

.  Capt.  Stoehrmann,  Kenneth  C.  ,  "New  Roles  for 
TAC's  F - 1 1 1 ,  "  Air  Force  Magazine,  Vol.  63, 
February  1980,  pp.  46-51. 

.  Ulsamer,  Edgar,  "EF-111A  (Tactical  Jamming 

System)  USAF's  Supersonic  Electronic  Shield,"  Ai i 
Force  Magazine,  Vol.  60,  August  1977,  pp.  54-59. 
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Sundaran,  G.S.,  "E-W  Dedicated  Aircraft  :  the 
EA-6B  and  EF-111A,"  International  Defense  Review 
10,  February  1977,  pp.  54-58. 


D.  MCLWG 

GAO  Reports 

.  GAO  Report  PSAD-78-122,  "An  Assessment  of  the 

Major  Caliber  Lightweight  Gun  System"  (SECRET), 
August  4,  1978. 

.  GAO  Report  PSAD-77-4,  "Operational  Test  inn  of  the 
Major  Caliber  Lightweight  Gun,"  November  5, 

1976  . 

E.  PATRIOT 


GAO  Reports 

.  GAO  Report  PSAD-73-15,  "Satus  of  the  Patriot 
Program"  (SECRET). 

.  GAO  Report  PSAD-76-89,  "SAM-D  Missile," 

February  17,  1976. 

Aviation  Week  and  Space  Technology 

"Brown  Approves  Limited  Patriot  Production," 
September  22,  1980,  p.  22. 

"Germans  Weighing  Alternative  Patriot  Acquisition 
Technique,"  August  29,  1977,  p.  47. 

.  Filter  Center:  System  capable  of  flying  target 
drones  in  formation  for  SAM-D  tests  to  be 
delivered,  May  31,  1976,  p.  51. 

.  Industry  Observer:  Marines  decide  SAM-D  is  too 
large  for  tactical  assault  mission,  May  3,  1976, 
p.  11. 


Arm-/  Presses  New  Missile,"  March  15  ,  1975,  p.  23. 


Melier,  Rud i ,  "The  U.  S.  Army's  Mobile 
Air-Defense  Missile  System  Nears  Production, 
International  Defense  Review  1 3 ,  Vo  1 .  4 , 
December  1980,  po.  495-499. 

Power,  Tom,  "Drones  Take  on  White  Sands 
Missiles,"  Air  Force  Times,  Vol.  40,  May  19, 
:>80,  p.  31. 
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Lt .  Col.  Naegele,  Eugene  L.,"  Automated  Operation 
With  Human  Control,  A  Key  Feature  of  Patriot," 

Air  Defense  Maqacine,  July-September  1978, 
pp.  17-23. 


.  _ ,  "DOD  Favors  Sur face-to-Surf ace  Version  of 

Patriot  for  Assault  Breaker  Role"  Armed  Forces 
Journal  International  Vol.  115,  August  1978, 
p.  13. 

.  _ ,  "Patriot,"  Signal,  Vol.  32,  April  1978, 

p.  71 . 

.  Lt.  Col.  Neagele,  Eugene  L.,  "Track  via  Missile 
Guidance,"  Air  Defense  Magazine,  Janua ry-Ma rch 
1978,  pp.  12-16. 

.  Lt.  Col.  Neagele,  Eugene  L.  "Multifunction  Phased 
Array  Radar,  a  Key  Feature  of  Patriot,"  Air 
Defense  Magazine,  October-December  1977, 
pp.  37-40. 

.  Capt.  Anderson,  R.E.,  "Patriot  -  Continuation  of 
a  Proud  Heritage,  Air  Defense  Magazine, 

April-June  1977,  pp.  32-36. 

ROLAND 

GAO  Reports 

.  GAO  Report  PSAD-79-91,  "Evaluation  of  the 
Decision  to  Begin  Production  of  the  Roland 
Missile  System"  (SECRET),  June  29,  1979. 
Unclassified  version  PSAD-79-100,  August  17,  1979 

.  GAO  Report  PSAD-79-28,  "How  Ready  Is  the  Army's 

Roland  Missile  for  Production?"  February  14,  1979 

.  GAO  Report  PSAD-78-128,  "Status  Report  on  the 
Roland  Missile  Program,"  July  7,  1978. 

.  GAO  Report  PSAD-73-16,  "Issues  To  Be  Resolved 
Before  Committing  the  Army's  Roland  Missile 
System  to  Production,"  March  10,  1978. 

.  GAO  Report  PSAD-77-38,  "Status  of  the  Roland  and 
Chaparral  Missile  Programs"  (CONFIDENTIAL)  , 
February  28,  1977. 

GAO  Report  PSAD-76-116,  "Roland  1 1 /Chapar ra 1/AAA / 
LOFADS  Systems,"  March  17,  1R76. 
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Aviation  Week  and  Space  Technology 

"U.S.  Falters  on  Commitments  to  NATO," 

November  10,  1980,  p.  35. 

"Tornado  Costs  Delay  Roland  2  Buy,"  October  20 
1980,  p.  113. 

"Roland  Demonstration  Tests  Scheduled," 

August  18,  1980,  p.  81. 

"Army  Limits  Roland  Production,"  January  14, 
1980,  p .  66. 

"Roland  Leaves  Launch  Tube  to  Intercept  Aerial 
Target,"  June  11,  1979,  p.  86. 

"Funding  Approval  for  Roland  May  Sour  Senate, 
House  Clash,"  June  4,  1979,  p.  21. 

"Roland  Proves  Feas  i  ~ :  1 1  f  at  Technology 
Transfer,"  September  11,  -  C  3 ,  p.  114. 

"3oeing  Tests  Roland,"  May  _2,  1973,  p.  25. 

"Roland  Tests  Start,"  F~briury  20,  I ‘^3,  p.  19 

"Roland  Effort  Tests  Techno l  vy/  Transfer, " 
November  14,  1977,  p.  13. 

"Roland  2  Nearing  U.S.  ^rodj/ti on,"  January  1“ 
1977,  p.  57. 

"Roland  Service  Inf  roo  Jet  ion  Nearly,"  May  3, 
1976,  p.  55. 

"Issue  of  Roland  Sal--'  r  y  C.S.  Unresolved , " 
January  12,  1976,  o.  9. 

her  Periodicals 

Robins,  Yves,  "Roland :  Answer  to  the  Low- Leve 
Air  Threat,"  N  A  TP's  F  i  f  t  a  e  n  :  ■'  a  t :  o  n  s  ,  Vol.  25, 
Feo rua ry-Ma rch  1980,  op.  60-66. 

_ ,  "Rapier  Buy,  Roland  Out?"  Armed  Forces 

Journal  International,  Vol.  117,  September 
p .  10 . 

_ _,  "Roland  Launched,"  Military  Review,  Vol. 

September  1979,  p.  76. 
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Geddes,  J.  Philip,  "Technology  Transfer  and  the 
U.S.  Roland  Missile,1'  Interavia,  Vo  1 .  3  3, 
November  1973,  pp.  1095-1099. 

S'nohat,  Murry,  "Production  Greased  for  Roland," 
Military  Electronics/Counter measures  ,  Vo  1 .  4 , 
September  1978,  p.  28. 

_ ,  "Roland:  Exchange  of  Expertise,"  National 

Defense ,  Vol.  62,  May-June  1978,  pp.  559-550. 

Brig.  Gen.  Ragano,  Frank  ?.,  "U.S.  Roland:  A 

Milestone  in  Technology  Transfer,"  Defense 
Management  Journal,  Vol.  14,  May  1978,  pp.  3-10. 

_ ,  "Roland:  Army's  Leading  Case-In-Point," 

Government  Executive,  Vol.  10,  March  1973,  p.  23 

_ ,  "U.S.  Roland:  A  Comprehensive  Update,"  Air 

Defense  Magazine,  Januarv-March  1978,  pp.  37-39. 

Richardson,  John  H.,  "Roland,  A  Technoloav 
Transfer  Program,"  Defense  Systems  M a n a g e m » n t 
Review,  Vol.  1,  Summer  1977,  pp.  13-17. 
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ADEQUACY  OF  T  &E  DATA 
-  Interview  Guide  - 


INTRODUCTION 


Booz ,  Allen  is  conducting  a  research  study  for  the  Defense 
System  Management  School  regarding  the  adequacy  of  test 
information  provided  at  the  time  of  program  milestone 
decisions.  It  has  been  observed  that  in  numerous  cases  the 
DSARC  has  rejected  the  services'  proposal  for  moving  into 
the  next  phase  or  has  attached  qualifications  involving 
additional  T&E  because  the  data  presented  to  them  is  "inadequate." 
To  date  no  one  has  been  able  to  precisely  define  what  constitutes 
adequacy  but  we  are  trying  to  characterize  the  scenarios  in 
which  this  phenomena  occurs  and  to  understand  the  reasons  for 
it.  We  hope  to  be  able  to  make  recommendations — perhaps  involv¬ 
ing  fundamental  changes  to  acqusition  policy — which  will  alleviate 
it  in  the  future.  Your  program  has  been  selected  for  detailed 
review  and  specifically  we  are  looking  at  the  DSARC  that  was 
held  on 
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1.  As  background,  we  are  trying  to  establish  a  decision 
profile  for  this  system.  Can  you  describe  the  program  plan 
as  of  the  previous  DSARC?  The  position  of  the  Service  going 
into  this  DSARC?  The  decision  resulting  from  this  DSARC? 
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2.  Was  there  an  approved  TEMP  at  the  time  o 
milestone  decision?  or  shortly  thereafter? 
If  there  wasn't,  why  not? 


the  previous 
( re  ference ) 


3.  Could  you  comment  on  your  view  of  the  utility  of  the  T 
(Probe  for  sense  of  contract  with  OSD  vs  useless  appendage 


4.  If  there  was  a  TEMP,  in  your  judgement  was  it  well  conceived? 
For  example,  did  it: 

Address  operational  suitability  as  well  as  technical 
performance 

.  Propose  evaluation  in  terms  of  actual  threat  in  a 
realistic  environment 

.  Require  typical  military  personnel  for  operation 
and  maintenance 

Provide  for  adequate  test  resources. 
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5.  Was  the  test  program  described  in  the  TEMP  (or  otherwise 
originally  planned)  carried  out?  If  not,  why  not?  For 
example  : 

.  Lack,  of  money 

.  Slips  due  to  technical  problems 

Lack  of  proper  test  faci lity/instrumentation 
Other  reasons. 
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6.  Could  you  describe  the  test  program  itself,  including: 

Were  requirements  well-defined  before  testing 
started? 

Did  requirements  change  after  testing  began? 

.  Were  adequate  resources  available?  (test  items, 
faci lities ,  etc.  ) 

.  Was  test  cost  a  factor?  or  did  testing  get  squeezed 
by  other  overruns  in  the  program? 

.  Other  comments  regarding  the  conduct  of  the  DT/OT 
phase . 
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7.  What  did  the  test  results  show?  What  was  the  position  of 
the  independent  evaluator? 

Issues  relate  primarily  to  DT  or  OT 
Failure  to  meet  thresholds 
.  Was  software  an  issue?  How? 


3.  If  there  was  a  case  of  "inadequate  data,"  what  is  your  view 
of  whv  it  occurred? 


9.  One  frequently  heard  complaint  is  that  the  items  being  tested 
in  IOT&E  are  not  sufficiently  mature  or  representative  of  the 
production  item  to  yeild  valid  data  DOD  policy  permits  limited 
production  of  items  for  operational  testing  prior  to  Milestone  III. 
Whv  wasn't  this  done? 


10.  Why  did  the  Service  go  to  the  DSARC  anyway?  (With  inadequate 
data.  ) 


11.  What  could  have  been  done  differently,  how  could  the  situation 
have  been  improved? 
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12.  In  your  opinion,  did  other  external  factors  influence 
the  decision  of  the  DS ARC?  For  example. 

Affordability  (either  in  near  or  far  term) 

.  Other  program  failures/controversies 
Recent  studies  (e.g.  GAO) 

Congressional  actions 

Dominant  personalities  within  OSD  (with  pet  theories) 
National  situation. 
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13.  Do  you  have  any  comments  on  reasonable  expectation  of  DSARC 
principles? 

.  How  much  is  enough?  (Confidence,,  sample  size,  etc.) 
Can  you  ever  have  too  much? 

How  should  OSD  integrate  the  total  body  of  data  and 
weigh  with  other  aspects  (e.g.  urgency)? 

What  should  the  role  be  of  simulations  and/or  analysis 
as  a  supplement  to  empirical  tests? 

.  How  should  one  treat  R&M  growth? 

What  was  the  impact  of  the  DDT&E  assessment? 

.  Was  there  an  awareness  of  the  cost  (resource 

implications)  of  decisions  to  delay  and  the  impact 
on  contracts  the  POM  process,  etc. 

.  Did  OSD  hold  up  there  side  of  the  "contract" 

regarding  goals  and  thresholds,  or  was  it  a  new  ball 
game? 
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OVERVIEW  OF  CHANGES  IN  ACQUISITION  POLICY 


This  Appendix  contains  a  brief  overview  of  some  of  the 
key  studies  and  other  events  since  1970  that  influenced 
acquisition  policy.  Specifically,  these  include: 

.  The  memoranda  to  the  Secretaries  of  the-  military 
departments  by  Deputy  Secretary  of  Defense  David 
Packard  in  1969  and  1570 

.  The  final  report  in  1972  of  the  Commission  or. 
Government  Procurement  (created  by  Congress  in 
1969)  which  eventually  led  to  the  creation  c£  the 
Office  of  Federal  Procurement  Policy  and  the  puc- 
lication  of  the  Office  of  Management  and  Budget 
(OMB)  Circular  A-109 

The  formation  of  tne  Acquisition  Advisory  Group 
(AAG)  by  Deputy  Secretary  of  Defense  William  ?. 
Clements  in  1975 

.  The  designation  of  the  first  Defense  Acquisition 
Executive  in  1976 

.  The  Defense  Science  Board  Summer  Study,  Report  of 
the  Acquisition  Cycle  Task  Force,  published  in 
March  1978 

The  Defense  Resource  Management  Study,  known  in¬ 
formally  as  the  Rice  Report,  published  in  Feb¬ 
ruary  1979  which  led  to  the  creation  of  tne  De¬ 
fense  Resources  Board. 

( 1 )  Packard  Initiatives 

The  Packard  memos  of  1969  and  1970  began  the 
modern  era  of  acquisition  reform  and  established  much 
cf  the  framework  for  the  acquisition  process  as  it 
exists  today.  Packard  saw  that  the  decentralization 
of  operations,  which  had  been  a  goal  of  the  previous 
administration  (along  with  centralized  decision 
authority  and  planning),  had  not  occurred.  He 
believed  that  OSD  should  be  removed  from  the  detailed 


A  comprehensive  overview  of  the  development  of  the 
defense  system  acquisition  process  is  given  in  "The 
Maturing  of  the  DOD  Acquisition  Process,"  by 
David  D.  Acker  in  the  Defense  Systems  Management 
Review,  Volume  2,  Number  3,  Summer  1980. 


direction  of  individual  defense  programs  and,  instead, 
that  OSD  should  assume  a  role  of  monitor  and  decision¬ 
maker  at  milestones  associated  with  major  system. 
Packard  offered  this  change  as  the  best  hope  of  im¬ 
proving  the  management  of  the  defense  system  acqui¬ 
sition  process  and  of  gaining  control  of  system  acqui¬ 
sition  cost.;.  Among  Packard's  initiatives  were  the 
following : 

.  Use  of  demonstrated  performance  as  the  pri¬ 
mary  measure  of  a  system's  readiness  to  pro¬ 
ceed  from  one  phase  of  the  acquisition  cycle 
to  the  next.  This  essentially  terminated 
the  Total  Package  Procurement  concept  which 
had  been  used  before. 

.  Institution  of  the  Development  Concept  Paper 
( DCP ,  later  the  Decision  Coordinating  Paper) 
as  a  "contract"  between  OSD  and  the  Services 
which  established  the  program  objectives, 
including  technical  requirements,  schedule 
and  cost,  and  the  thresholds  which,  if 
breached,  would  trigger  a  review  of  the  en¬ 
tire  program. 

.  Establishment  of  the  DSARC  as  an  advisory 
body  to  the  Secretary  of  Defense  on  the 
status  and  readiness  of  major  system  to  pro¬ 
gress  to  the  next  acquisition  phase. 

The  DSARC  as  originally  conceived  was  intended  to 
complement  the  DCP  approval  process  by  providing  a 
means  for  coordinated  review  and  evaluation  by  the 
senior  OSD  staff.  Thus,  the  original  purpose  of  the 
DSARC  was  as  much  to  inform  the  staff  of  the  status  of 
designated  programs  as  it  was  to  have  them  sit  in 
judgment  of  it.  This  was  consistent  with  Packard's 
desire  to  disengage  OSD  from  the  day-to-day  operation 
of  individual  programs. 

Other  changes  by  Packarl  included  strict  avoid¬ 
ance  of  unnecessary  concurrency  between  development 
and  production;  emphasis  on  hardware  competition  and 
prototypes,  rather  than  analysis  ("paper  studies"),  as 
a  means  of  judging  the  value  of,  and  selecting  be¬ 
tween,  alternative  weapon  systems;  and  clarification 
of  responsibilities  of  OSD,  the  military  departments, 
and  the  proaram  manager.  Also,  Packard  addressed  the 
problem  of  cost  growth  in  defense  systems.  One  cf  his 
r ecomme ndations  eventually  led  to  the  formation  in 

of  the  Cost  Analysis  Improvement  Group  (CAIG)  as 
unotner  input  to  the  DSARC  milestone  review  process. 


In  the  1969  time  frame,  the  President  and  the 
Secretary  of  Defense  chartered  a  Blue  RibDon  Defense 
Panel  to  study  the  entire  organization,  structure,  and 
operation  of  DOD.  Many  of  the  resulting  findings  were 
directly  related  to  defense  system  acquisition.  Sec¬ 
retary  Packard  used  the  Blue  Ribbon  Defense  Panel  re¬ 
port  and  his  own  experience  to  formulate  a  cohesive 
statement  of  DOD  acquisition  policy  in  the  form  of  the 
first  DOD  Directive  5000.1,  Acquisition  of  Major 
Systems,  in  July  1971. 

(2)  Commission  on  Government  Procurement 


During  this  same  period,  Congress  was  exhic : t : no 
.ncreased  dissatisfaction  with  the  entire  Federal  pro¬ 
curement  process.  In  November  1969,  Public  Law  91-729 
was  enacted  which  established  the  Commission  on  Gov¬ 
ernment  Procurement  (COGP)  to  make  recommendations  on 
improving  the  effectiveness  and  efficiency  of  Federal 
procurement  practices.  After  nearly  3  years  and  an 
exhaustive  series  of  hearings,  the  Commission  pub¬ 
lished  its  report  in  December  1972.  It  made  149 
sweeping  recommendations,  12  of  which  concerned  the 
system  acquisition  process.  These  recommendations 
were  focused  primarily  on  the  "front  end”  of  the  pro¬ 
cess  where  the  Commission  found  that  most  problems 
began.  The  recommendations  were  grouped  in  fiv°  major 
categories  : 

.  Establishing  needs  and  goals 

.  Exploring  alternative  systems 

.  Choosing  a  preferred  system 

.  System  implementation 

.  Organization,  management,  and  personnel. 

The  recommendations  were  generally  well  received 
and  independently  supported  by  the  DOD  Acquisition 
Advisory  Group  (described  below).  in  April  1976,  the 
Office  of  Federal  Procurement  Policy  (which  was  itself 
an  outgrowth  of  the  COGP,  having  been  established  by 
Congress  in  late  1974  pursuant  to  P.L.  93-400)  issued 
GMB  Circular  A-109  which  embodied  the  acquisition  phi¬ 
losophy  that  had  emerged.  A-109  articulated  a  system 
approach  to  acquisition  which  combined  problem  solving 
logic,  increased  flexibility,  and  several  fundamental 
management  and  business  principles.  The  management 
principles  involved: 

.  Establishing  clear  responsibilities  and 

authorities  for  the  program  manaoer 

.  Generating  a  tailored  acquisition  strategy 

early  in  the  program  to  guide  the  entire 
effort 
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.  Requiring  approval  by  the  agency  head  for 

continuing  the  program  at  key  decision  points 

Encouraging  creative  new  ideas  by  avoiding 
government  specified  solutions  t^  defined 
problems . 

The  business  principles  included: 

.  Creating  and  maintaining  competition  as  far 
along  the  process  as  is  economically  bene- 
f  icial 

.  Continuing  contractor  work  without  inter¬ 
ruption  as  the  Government  goes  about  its 
decisionmaking 

.  Recognizing  that  the  Government  must  pay  for 
the  benefits  that  result  from  increased 
competition. 

The  DOD  has  gradually  adopted  the  precepts  of 
A-109.  Revisions  to  DOD  Directive  5000.1  and  DOD  In¬ 
struction  5000.2  have  incorporated  a  new  milestone  0 
and  emphasized  the  front-end  mission  analysis  by  re¬ 
quiring  a  mission  element  need  statement  (MENS). 

While  there  have  been  critics  who  attribute  increased 
time  and  cost  to  A-109  policies,  the  present  philos¬ 
ophy  appears  to  be  that  the  policies  themselves  are 
sound,  and  that  improvements  are  to  be  made  by  better 
implementation  of  the  fundamental  principles. 

( 3 )  Acquisition  Advisory  Group 

In  1975,  Secretary  Clements  created  the  AAG  to 
review  the  separate  recommendations  of  the  three 
Services  on  improving  the  acquisition  process.  The 
AAG  found  that  the  existing  policies  as  stated  in  DOD 
Directive  5000.1  were  sound  but  that  improvements  were 
needed  in  the  way  the  policies  were  implemented  by 
both  OSD  and  the  Services.  In  particular,  the  AAG 
stressed  decentralization  of  the  management  of  indi¬ 
vidual  programs  and  a  stricter  interpretation  of  the 
DS ARC  role  in  providing  advice  to  OSD  on  designated 
major  systems  only  at  formal  milestone  points.  By  and 
large,  the  recommends tions  of  the  AAG  were  consistent 
with  those  of  the  COGP  and  supported  the  management 
thinking  that  led  to  A-109. 
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Responding  to  A-109,  the  DoD  appointed  its  first 
Defense  Acquisition  Executive  (DAE)  in  1976.  The  DAE 
was  to  be  the  primary  advisor  to  the  Secretary  of 
Defense  on  matters  relating  to  the  acquisition  of  de¬ 
fense  systems.  The  significance  of  the  appointment 
lay  in  the  fact  that  previously  there  was  a  strong 
distinction  between  the  R&D  community,  which  was  the 
responsibility  of  the  DDR&E,  and  the  production  com¬ 
munity,  primarily  the  purview  of  the  Assistant  Secre¬ 
tary  of  Defense  (Installations  and  Logistics) 
(ASD(I&L)).  Because  the  two  communities  viewed  the 
process  differently,  participated  to  different  degrees 
at  different  times,  and  generally  did  business  in  dif¬ 
ferent  ways,  there  was  a  cultural  gap  between  them 
which  compounded  the  problems  of  transitioning  between 
R&D  and  production.  The  DAE's  responsibility  was  to 
integrate  and  unify  the  management  process;  however, 
there  are  some  indications  that  vestiges  of  divided 
cognizance  still  exist. 

( 5 )  Acquisition  Task  Force 

A  somewhat  contrary  note  was  sounded  in 
March  1978  with  the  publication  of  the  report  of  the 
Acquisition  Cycle  Task  Force  of  the  Defense  Science 
Board.  The  Task  Force  examined  a  number  of  major 
systems  acquisition  case  histories,  changes  in  acqui¬ 
sition  policy  over  the  past  two  decades,  and  the  act¬ 
ual  workings  of  the  program  advocacy  and  budgetary 
processes.  They  concluded  that,  over  the  past  15  to 
20  years,  the  acquisition  process- -f rom  initial  pro¬ 
gram  concept  to  initial  operational  capability--had 
increased  in  length  by  approximately  5  years  on  the 
average.  Another  conclusion  was  that  most  of  this 
increase  had  been  in  the  front  end  of  the  process, 
i.e.,  from  the  time  of  initial  concept  studies  to  the 
start  of  Full-Scale  Development  (FSD)  and  at  the  back 
end  from  the  completion  of  FSD  to  the  achievement  of  a 
full  operational  capability. 


The  Task  Force  also  concluded  that  full-scale 
prototyping  often  introduced  costly  and  unnecessary 
program  delays,  that  the  policy  of  "no  concurrency" 
was  being  applied  too  rigidly  and  was  inefficient  and 
costly  in  many  cases,  and  that  sequential  and  separate 
testing  of  systems  by  developers  and  users  (instead  of 
the  independent  evaluation  of  jointly  obtained  test 
data)  often  added  unnecessary  expense  and  time.  It 
was  also  concluded  that  the  practice  of  incremental 


dec l s ionmak i ng - -by  requiring  the  achievement  cf  a  con¬ 
sensus  between  the  advocates  and  the  antagonists  at 
each  mi les tone- -was  a  cumbersome  process  that  tended 
to  prolong  programs  so  much  that  the  threat,  the 
players,  and  even  the  concepts  of  warfare  often 
changed  during  the  acquisition  cycle.  This  resulted 
in  canceling  programs  and  starting  new  ones  to  meet 
the  threat  with  the  concomitant  waste  of  limited 
national  resources. 

Finally,  the  Task  Force  expressed  concern  that 
without  very  careful  attention  to  the  implementation 
of  OMB  Circular  A-109,  procedures  could  be  insti¬ 
tutionalized  that  would  significantly  increase  the 
total  length  of  the  acquisition  process  without  achie¬ 
ving  the  intended  degree  of  program  flexibility  and 
stability  in  acqu isi t ion  policy. 

The  DSB  made  recommendations  regarding  afford¬ 
ability  of  new  weapon  systems,  flexibility  of  the 
acquisition  process,  tailoring  of  phases  and  mile¬ 
stones,  limiting  phototyping,  increasing  concurrency 
when  warranted,  combining  development  and  operational 
testing,  and  coupling  the  DSARC  and  PPBS.  In 
addition,  it  suggested  that  product  improvement  cf 
existing  system  be  considered  as  an  alternative  to  new 
system  development  whenever  possible. 

Some  acquisition  reformists  felt  that  the  DSB 
report  overly  favored  expediency  as  an  operating 
mode.  Nevertheless,  most  of  their  recommendations 
were  reflected  in  the  March  1980  revisions  to  500C.1 
and  5000.2.  As  a  result,  there  was  a  shift  m  the 
swing  of  the  pendulum  away  from  excess  caution  toward 
greater  risk-takinq. 

One  of  the  more  important  cnar.gea  of  these  latest 
revisions  to  5000.  1  and  " 0 0 C . C  in  t«rm s  of  under¬ 
standing  the  decision  pr  •><>>.:.-  : tr.it  milestones  0  and 
II  were  given  important  ir.tr.  Vilest  're  0  now- 

re  fleets  a  commitment  ir.  *  r»-  .  art  .  r  ~:e  Secretary  of 
Defense  to  "satisfy  tr.e  •  '  m  i  mi  1  ester.-  II  ms 
intention  to  "deploy  trm-  .•/.-*.*  m. . "  I:  •>  siimricance  of 
the  latter  is  that  it  :  r  r.  -w  ■  ‘f:  •:  » 1  y  uc Knowledges 
that  the  major  commitment  -  * :  o  p  r  r-<i  r  air  is  really  to 

be  made  at  milestone  II  and  not  at  m 1 1  III  as 


Exhibit  1 1 1-1,  taxen  from  tr.e  Boos,  Mien  wall  chart, 
"Acquisition  Process  for  V.a  Mr  Defense  Systems,"  Feb¬ 
ruary  1981,  discusses  tn«*  full  range  of  policy  changes 
that  were  thus  intro(iucrd. 
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MARCH  1980  CHANGES  IN  POLICY 


On  March  19  1980.  (hi  Department  of  Defense  extensively  revised  the  articulation  ot  its  top  level  policy  end  pro 
codurel  framework  tor  tho  acquisition  ot  motor  systems.  DOD  Directives  8000.1  end  80D0.2  -  lest  revised  end  issued 
on  Jenusry  18,  1977  lor  the  purpose  of  implementing  QMS  Circuler  A  109  -  wore  concollod  end  replsced  by  OOO 
Oirective  5000  1  "Motor  System  Acquisitions'  end  000  Instruction  8000.2.  Motor  System  Acquisition  Procedures.  Tho 
lollowmg  is  e  listing  ot  some  ot  the  more  imporlent  chenges  in  policy  (including  emphesisi  end  m  procedures  es  retlec 
ted  m  the  current  revisions: 

•A  109.  There  is  e  ter  stronger  commitment  to  emhreemg  tho  fundementol  precepts  ot  OMR  Circuler  A  109  The  circu 
lor  is  now  enclosed  with  DOD  Directive  8000.1  Irethor  then  roleroncodl  end  e  number  ot  its  key  obiectives  sro  am 
phasired  throughout  both  8000.1  end  8000.2  which  ere  estebltshed  es  hevmg  the  highest  precedence  over  ell  other 
000  acquisition  related  issuences. 

•  AFfORQABILITY  The  concept  ot  affordability"  -  the  overell  reconciliation  ot  resources  end  needs  -  hes  been  emphe 
sized  end  mode  e  meter  determinent  in  the  decision  making  process.  At  oech  ot  the  decision  milestones  tho  000  Com 
ponont  must  provide  essurence  thet  resources  ere  eveileble  or  con  be  progrsmmed  to  execute  tho  progrem  in  on  etti 
ciont  end  effective  mennor. 

•  PPBS  INTERFACE.  Tho  persistent  problem  ot  better  interface  between  scquisition  milestone  decisions  end  ennuol  PPBS 
overell  resource  allocution  decisions  hes  boon  given  mcreesed  emphssis.  In  psrticular.  the  USDRE.  ASOICI  end  the  ASO 
IPA&EI  ere  given  bridging  responsibility  tor  reviewing  effordebility  issues  le.g..  cost,  priority,  end  eveilobility  ot  fiscal 
and  manpower  ressurcasi  in  the  context  o<  the  PPBS  process. 

•  MILESTONE  DECISION  COMMITMENT.  Decisions  by  tho  SECDEF  et  Milestones  0  end  II  have  been  given  importont  now 
meanings.  Previously,  tho  Milestone  0  decision  represented  a  commitment  only  to  identify  and  explore  alternative  soli/ 
tions  and  tho  Milestone  It  decision  represented  a  commitment  only  to  continue  the  program  through  tho  engineering 
development  phase.  Tho  decisions  now  reflect  e  commitment  on  the  part  ot  tho  SECDEF  to  "satiety  the  need 
(Milestone  01  and  to  "deploy  tho  system"  (Milestone  III. 

•  MAJOR  SYSTEMS  DESIGNATION.  Rules  tor  designating  system  as  "meior"  have  been  relaxed  in  favor  of  more  pregmo 
tic  guidelines.  Rules  romein  tor  determining  when  a  MENS  is  required  (I100M  R0T&E  or  S800M  procurement)  but  the 
resulting  system  may  or  may  not  bo  designated  es  meior.  This  is  consistent  with  the  often  stated  obiective  ot 
reducing  ihe  number  of  systems  going  through  the  lormei  DSARC  process. 

•  TIMELINESS.  TAILORING  AN0  CONCURRENCY.  The  concept  ot  "tailoring"  tho  acquisition  process,  ss  described  in  0MB 
Circuler  A  109.  is  not  now:  however,  the  concept  of  planned  "concurrency."  expressly  proscribed  during  the  Packard 
era.  is  cautiously  reintroduced  as  a  means  ot  minimizing  acquisition  time  when  technical,  cost  and  supporlability  risks 
are  low  or  when  urgency  to  counter  a  threat  is  peremount.  Examples  cited  include:  increasing  funding,  overlapping, 
combining,  or  omitting  acquisition  phases  or  overlapping  or  combining  development  T&E  with  operationol  T&E  Also,  the 
concept  of  accepting  system  performance  growth  after  deployment  is  included  as  a  means  ot  achieving  improved  re 
sponsiveness  to  the  threat. 
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EXHIBIT  C-i  (Continued) 


MARCH  1980  CHANGES  IN  POLICY  (con't) 


•  DEFENSE  SYSTEMS  ACQUISITION  REVIEW  COUNCIL.  OSARC  membership  has  been  reconstituted:  tha  Undar  Secretary  ol 

Dalansa  for  Policy  and  Ilia  Chairman,  Joint  Chiati  ol  Staff  have  baan  addad  aa  parmanant  mambara;  tha  ASD  IC’III 
haa  baan  mada  a  principal  adviaor.  Iratlacting  an  organizational  change  and  replacing  tha  ASDII)  and  Director.  Telecom 
mumcationa  who  ware  previoualy  mambara!;  an  0S0  officiate  have  baan  addad  aa  principal  adyiaora:  and  tha  Service 
Secretariea  whoaa  programa  ara  under  review  have  baan  eliminated  aa  daaignatad  participanta.  Further,  mileatone 
daemon  making  and  review  procadurec  hava  baan  rafinad.  OSD  raaponaibilitiaa  clarified  and  new  information  and  docu¬ 
mentation  requiremanta  lauch  aa  tha  IPS.  MRF  and  SDDMI  hava  bean  impoaad.  la  order  to  diaciplina  tha  procaaa.  a 

rigid  8  month  timetable  lor  ataffing  haa  baan  addad.  Saa  DSARC  Mileatone  Planning  Schaduia  at  bottom. 

•  INTERACTIVE  THREAT  ANALYSIS.  Tha  bay  interaction  between  intelligence  collection  and  weapon  ayatam  performance 

and  daaign  haa  for  tha  firat  time  formally  baan  mada  a  part  of  tha  acquisition  procaaa.  A  continuing  "interactive  anal 
yaia  "  -  a  atudy  of  tha  system-throat  interaction  -  ia  now  required  with  tha  firat  aubmiaaion  to  be  completed  before 
Mileatone  I.  Tha  raault  of  thia  procaaa  ia  a  aat  of  Critical  Intelligence  Parameters  (CIPal  which  ara  controlled  and  up¬ 
dated  throughout  tha  lila  of  tha  program.  00D  Components  ara  required  to  identify  CIPa  in  ordar  to  guide  collection 
activities,  and  DIA  is  required  to  validate  threat  data,  review  CIPs  output  and  report  findings  and  concluaiona  to  tha 
DAE  prior  to  DSARC  maatinga. 

•  ACQUISITION  STRATEGY  FOR  C1  SYSTEMS.  Certain  command  and  control  systems  hava  baan  formally  recognized  aa 

differing  from  other  weapon  systems  and  aa  having  ma|or  characteristics  which  require  special  management  procedures. 
For  such  systems,  an  acquisition  strategy  is  suggested  which  allows  aarly  implementation  and  field  evaluation  of  a 
prototype  using  oaistmg  commercial  or  military  hardware  or  software.  This  permits  design  and  testing  in  an 

ovolutionary  manner.  In  such  a  strategy  the  Concept  Eaploration  and  Demonstration  and  Validation  phases  normally  are 
combined  and.  in  some  cases,  procurement  for  initial  fielding  may  proceed  without  a  Full-Scale  Development  phase. 
Since  tailoring  of  the  acquisition  process  is  an  inherent  part  of  developing  an  acquiaition  strategy,  it  is  not  clear  why 
C1  systems  have  been  separately  highlighted.  A  mors  significant  distinction  is  that  in  C1  systems  the  designated  User  is 
now  tasked  to  perform  operational  tasting  Irathar  than  an  independent  operational  tester!  and  haa  been  assigned  the 
major  responsibility  lor  the  Demonstration  and  Validation  phase  Irathar  than  the  program  manageri. 

•  NATO  RSI.  Consideration  of  NATO  doctrine,  threat  aaaasament  and  mission  needs  and  the  standardization  or  mtoropora 
bility  of  U.S.  oquipmont  with  that  of  other  members  of  NATO  las  required  by  the  "Culver  Nunn  Amendments  ")  remains 
a  tenet  of  the  directives.  However,  the  revisions  to  DODI  5000.2  significantly  increase  opportunities  for  NATO  country 
participation  in  the  U.S.  acquisition  process  by  requiring  that  consideration  be  given  to  all  esisting  and  developmental 
NATO  systems  as  an  alternative  to  U.S.  development  and  that,  under  soma  conditions.  NATO  member  contractors  be 
solicited  for  bids  and  proposals  on  U.S.  systems  and  components.  In  addition,  international  codevelopmont.  coproduction, 
and  cooperative  testing  is  omphasized. 

•  MANPOWER.  TRAINING  AND  LOGISTICS  SUPPORT  ABILITY.  The  current  Directive  and  Instruction  further  emphasize  that 
logistics  supportability  is  a  design  requirement  as  important  as  cost  schedule  and  performance.  They  direct  that  man 
power  and  personnel  factors,  suen  as  numbers,  occupations,  and  skill  levels  required,  be  included  as  considerations  and 
constraints  in  system  design  and  that  aerly  in  the  acquisition  process  manpower  requirements  be  subiacted  to  trade 
offs  with  system  characteristics  and  support  concepts  in  order  to  achieve  the  most  cost-effective  balance  between 
acquisition  and  ownership  costs  and  system  effectiveness. 

•  ROLE  OF  GOVERNMENT  LABORATORIES.  The  formal  role  of  the  laboratories  in  the  acquisition  process  has  been 
diminished.  Formerly,  the  program  manager  was  required  to  engage  a  D00  laboratory  or  tha  purpose  of  evaluating 
risks  and  preparing  a  Tochnology  Assessment  Annes  to  the  DCP.  While  formal  risk  assessment  remains  an  inherent 
part  of  the  acquisition  process,  ths  former  requirement  has  been  eliminated  end  the  r  ale  ol  the  laboratories  is  left  to 
the  services. 
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qenerally  believed.  The  DSB  called  for  establishment 
of  milestone  III  as  the  decision  for  rate  production, 
reflecting  their  concern  that  programs  were  being  un¬ 
necessarily  stretched-out  by  limited  production 
phases.  Actually,  the  fact  that  milestone  II  includes 
the  authority  for  limited  production  of  hardware  for 
operational  test  and  evaluation  is  stated  in  A-109  and 
has  been  DOD  policy  since  1977,  but  it  is  not  commonly 
understood  or  followed. 

( 6)  Defense  Resource  Management  Study 

When  President  Carter  and  Secretary  of  Defense 
Brown  took  office  in  1977,  they  commissioned  the  De¬ 
fense  Resource  Management  Study  (DRMS) .  This  study 
was  to  address  the  resource  allocation  process  and 
four  of  its  specific  functions  including  weapon 
systems  acquisition.  (Other  functions  were  logistic 
support,  enlisted  personnel  management,  and  military 
health  care.)  The  DRMS  proposed  significant  changes 
in  the  PPBS  including  the  establishment  of  a  Defense 
Resources  Board  (DRB)  to  manage  a  combined 
program/budget  review  during  each  POM  cycle.  Secre¬ 
tary  Brown  established  the  DRB  in  March  of  1979,  and 
subsequently  changed  the  membership  of  the  DSARC  to 
coincide  with  that  of  the  DRB.  Thus,  he  attempted  tc 
achieve  greater  coupling  between  the  DSARC  and  PPBS 
processes . 

With  respect  to  acquisition  policy,  the  DRMS  of¬ 
fered  suggestions  on  t.ne  implementation  of  A-1C9, 
stressed  competition  in  the  acquisition  process,  and 
observed  that  "Troublesome  and  costly  problems  often 
arise  from  the  premature  commitment  of  systems  to 
high-rate  production."  The  DRMS  recommended  the  delay 
of  high-rate  production  until  both  technical  adequacy 
and  operational  suitability  (including  reliability, 
supportability,  and  readiness  characteristics)  were 
demonstrated . 

This  DRMS  recommendation  is  highlighted  because 
it  reflected  a  growing  concern  during  the  later  1970s 
regarding  deficiencies  in  the  operational  suitability 
of  U.S.  military  equipment.  Championed  by  new  per¬ 
sonnel  within  OSD,  operational  suitability  increas¬ 
ingly  became  a  major  topic  in  DSARC  reviews.  In 
January  1980  the  Office  of  the  Under  Secretary  of  De¬ 
fense,  Research  and  Engineering  (OUSDR&E) /DDT&E  and 
Office  of  the  Assistant  Secretary  of  Defense 
(Manpower,  Reserve  Affairs,  and  Logistics) 

(OASD (MRA&L) )  jointly  published  the  Operational  Suit¬ 
ability  Verification  Study  which  attempted  to  improve 
operational  suitability  evaluation,  particularly  as  it 
applied  to  the  test  community  and  users  of  test  data. 
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CHI  SQUARE  TEST  OF  INDEPENDENCE 


CHI  SQUARE  TEST  OF  INDEPENDENCE 


One  way  to  test  the  statistical  relationship  between 
the  type  of  DSARC  review  and  the  likelihood  of  unconditional 
approval  is  by  application  of  the  chi  square  test  of  inde¬ 
pendence.  Shown  below  is  a  2  x  2  matrix,  commonly  referred 
to  as  a  contingency  table,  which  describes  the  data  pre- 


The  chi  square  procedure  permits  a  judgement  as  to 
whether  there  is  an  interaction  between  the  type  of  review 
and  the  type  of  decision.  In  this  case  the  null  hypothesis, 
Ho,  is  that  there  is  no  relationship  between  the  two;  i.e., 
programs  are  just  as  likely  to  receive  unconditional  approval 
at  Milestone  II  as  they  are  at  Milestone  III.  This  hypothesis 
is  rejected  only  if;  ^  2  >  ^  2  where: 

0  a  ,  d. f . 


and  0 j_ j  are  the  observed  frequency  for  the  ith  row  and  jth 
column,  and  Ej_j  the  expected  frequency  (based  on  pure  chance) 
for  the  same  row  and  column.  Alpha  ( a )  is  the  confidence 
limit  of  the  test,  and  d.f.  is  the  number  of  degrees  of  free¬ 
dom,  which  in  this  case  is  one. 
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